Performance

In November 2022, we conducted load testing of the Bitrix24 Self-hosted (Enterprise edition) with a deployed CRM solution to assess its performance.

General information

During the load test, we deployed a CRM system in a clustered solution, filled it with a large amount of test data and subjected it to a load that imitated the behavior of real-life users. The operations performed for load testing purposes included standard CRM scenarios: system authorization, customer base handling, lead management, deal processing and deal status updating, instant messaging and document uploading.
Based on the needs of large-scale corporate segment, we have determined the key objectives for such testing:

• simulate the operation of a CRM system deployed at a large corporation (tens of thousands of employees, thousand CRM users + large amount of demo data, with high intensity information flowrates)
• provide a testing methodology that’s as close as possible to user behavior in real life
• demonstrate the stable operation of inbuilt Bitrix24 scaling and fail-safety technologies on available equipment
• ensure the response time of deployed solution does not exceed 1 second.

Test Conditions

Equipment

Dedicated virtual servers were selected for performing the test. This allowed to emulate a full-scale cluster solution with simultaneous overhead reduction for infrastructure deployment and support. Similar to the previous testing, we have employed equipment from the mid-price segment available on the market.
The hardware provided for the test featured three configurations of servers:

• Database server (2 pcs.): Intel Xeon (Skylake, IBRS): 2294 MHz (10 cores), 48 GB RAM, 300 GB HDD
• Application Server (2 pcs.): Intel Core (Haswell, noTSX, IBRS): 1995 MHz (6 core), 24 GB RAM, 200 GB HDD
• Load generator (1 pcs.): Intel Xeon (Skylake, IBRS): 2294 MHz (10 core), 24 GB RAM, 300 GB HDD.

Cluster solution and web environment

pfweb1, pfweb2	Cluster featuring two application servers (web servers): CentOS 7.9, Nginx 1.20.2, Apache 2.4.6, PHP 8.0.19. Balancing was performed using Nginx at pfweb1
pfdb1, pfdb2	Cluster featuring two database servers: CentOS 7.9, Percona Server (MySQL) 8.0.29, Master / Slave configuration.
pfjm	Load generating server with Jmeter 5.3.3 InfluxDB for Jmeter and Grafana data retrieval and storage to test result display and visualization
data processes and threads (1-7)	Test traffic Load balancing Cluster cache Cluster sessions File synchronization between Lsync nodes Database replication Core database connections for product performance

We have deployed a cluster featuring 2 database servers and 2 application servers. This specific configuration is standard and most widespread, chosen to ensure high cluster performance while also providing high availability.
Server software was configured using the Bitrix24 Virtual Appliance for Linux 7.5.2 and the Web Cluster module enabled to create a cluster solution.
After conducting initial "trial" tests, server configurations were updated.

List of web environment updates:

• updated operation system parameters:
  • echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
  • sysctl -w net.ipv4.tcp_tw_recycle=1
  • sysctl -w net.ipv4.tcp_tw_reuse=1
• sync_binlog increased to 1000
• innodb_buffer_pool_size increased to 34 GB
• max_heap_table_size set as 256 MB
• max_heap_table_size, max_heap_table_size set as 90 MB
• myisam_sort_buffer_size increased to 64 MB
• max_connections increased to 305

Intranet portal test solution

Cluster solution had a deployed Bitrix24 Self-hosted (Enterprise edition), version 22.375 (Main module version).

1. Standard Bitrix24 Self-hosted (Enterprise edition), version 22.375 with the latest updates and the Web Cluster module enabled to create a cluster solution.
2. Demo content at the start of the final test:
   • 500,000 companies
   • 1,515,000 contacts
   • 995,000 leads
   • 500,000 deals
   • 144,000 catalog products
   • 9,391,000 CRM events
   • 4,522,000 CRM timeline records
   • 36,000 Feed messages
3. The number of employees in the portal database at the time of the start of the final test is 22,198 (distributed across 14 structural divisions).

Load generation method

Testing was performed using two main CRM scenarios:

• The "Quick leads" scenario with a large number of managers (up to 500 and more) that manage the flow of incoming leads for subsequent lead qualification as lost or won/conversion to a deal for purchase of products/services.
• The "Longterm deals" scenario with a large number of managers (up to 100 and more) that handle deals, limited per 1 manager. Each day, a manager adds new statuses and updates to handled deals: schedules meetings, uploads documents, comments, calls and corrects specification. Key feature: large number of products in an individual deal.

The test envisaged the company using both scenarios simultaneously, associated with several business directions.
Selected load scenario was quite resource-consuming, allowing to check performance for both Bitrix24 portal and CRM feature within the framework of complex projects.

Test scenarios details are provided below:

Scenario (load %)	Action	Weight, %
Quick leads (80%)	Authorization	100
	Lead list	100
	Repeated lead list	100
	Lead	100
	Product upload	100
	Product upload 2	100
	Lost lead	18
	Won lead	2
	Update of kanban, leads, push	100
	Lead list	100
Authorization (20%)	Authorization	100
	Lead list	100
	Deal list	100
	Opened deal	100
	Payment document upload	100
	Adding a call	5
Companies (1%)	Authorization	100
	Lead list	100
	Company list	100

The load was created using the JMeter tool, version 5.5. The test data was recorded via InfluxDB, a high-performance database designed to handle high query and record loads. Analysts used Grafana for visualization. The servers were monitored using the Telegraf application.

Load generation method

Bitrix24 CRM (Enterprise edition) performing within test scenarios emulating a CRM-based solution for a large company, deployed on a cluster of 4 physical servers in a 24-hour test has ensured stable concurrent CRM operation for 3000 employees.
The monitoring system demonstrated 423 874 system queries per 24-hours with average response time 0.713 seconds in 95% of requests.

Test summary: number of flows, dynamics for number of queries per second, total number of queries and errors:

Response time. 95 percentiles for specific pages and their traffic within a test:

CPU use at database and app servers:

Conclusion

The results of the load test of the Bitrix24 Self-hosted (Enterprise edition) confirmed the high performance of the platform and stability in high load conditions. The demo CRM solution, deployed in a cluster solution of 4 servers ensured the simultaneous operation of 3000 users, which corresponds to the approximate load profile for a large corporation.

At the same time, the system provided fast (even by the standards of Internet projects) response not exceeding 1 second for the majority of requests, which ensures a smooth user experience. The web cluster technology has once again proven its performance and reliability.

The testing methodology, scenarios, and load profiles as well as the use of physical user profiles allowed us to get the test conditions that were as close to real life as possible. This means that the results of this test can be used as a reference for designing new projects.

We are ready to provide the testing methodology, portal configuration, and test environment upon request to our corporate customers and integrators to serve as a basis for conducting full-scale testing of custom-developed portals or their prototypes, testing hypotheses, calculating the safety factor of our solution under peak loads. The test environment can be further customized - e.g., by adding integrations with a company’s corporate systems to ensure more accurate results.

Please, contact us to obtain any additional testing details.

In January 2021, we conducted load testing of the self-hosted version of Bitrix24 (Enterprise edition) to assess the performance of a large company’s intranet portal built on its basis.

General information

During the load test, an intranet portal deployed in a clustered solution and filled with a large amount of test data was subjected to a load that imitated the behavior of real-life users.

Virtual users performed sets of operations that resembled typical work scenarios:

• logging in/out
• reading news
• opening workgroups
• browsing the Feed
• creating individual and group chats
• exchanging instant messages
• setting tasks
• downloading documents
• launching business processes

Each user completed one of the typical scenarios and finished their work on the portal.

Test objectives

• simulate the operation of an intranet portal of a large corporation (100,000 employees + a large amount of demo data)
• provide a testing methodology that’s as close as possible to user behavior in real life
• demonstrate the stable operation of the portal without errors on available equipment (in a situation when ⅓ of all company’s employees (at least 30,000) are using the portal simultaneously)
• ensure the response time of the portal does not exceed 1 second (for 95% of requests)

Test Conditions

Equipment

The hardware provided for the test featured physical servers in two configurations:

• Database server: Intel Xeon W-2255 3.7 GHz (10 cores), 128 GB DDR4, 2 x 960 GB NVMe + 2 x 8000 GB HDD
• Application Server: Intel Xeon E-2236 3.4 GHz (6 cores), 32 GB DDR4, 2 x 480 GB SSD

A cluster featuring 2 database servers and 3 application servers (web servers) was deployed. This particular configuration was chosen to ensure high cluster performance while also providing high availability.

Test intranet portal parameters

1. Standard self-hosted version Bitrix24 (Enterprise edition), version 20.x.x with the latest updates and the Web Cluster module enabled to create a cluster solution.

2. Demo content at the start of the final test:

• 590,000 messages in the Feed
• 540,000comments
• 40,000 news posts
• 180,000 tasks
• 415,000 instant messages

3. The number of employees in the portal database at the time of the start of the final test is 111,304 (distributed across 67 structural divisions).

Load generation method

The load was created using the JMeter tool, version 5.3.3. The test data was recorded via InfluxDB, a high-performance database designed to handle high query and record loads. Analysts used Grafana for visualization. The servers were monitored using the Zabbix monitoring system.

For testing purposes, 29 scenarios were selected from 13 blocks characteristic of a typical intranet portal:

• Authorization
• Feed
• Search
• Chats (group and one-on-one)
• Tasks
• Calendar
• Drive
• News posts
• Picture gallery
• Employees
• Profile
• Business processes
• Workgroups

For each test, a special load distribution system was used taking into account the work of various users on the portal and the share of each of the blocks in the total load.

Unlike the previous load test in 2015, we used a new method of generating user load. Instead of one virtual user randomly performing sets of standard actions, the load was generated by a large number of different users each with their own account on the portal.

The load generator logged each of these users into the system and then executed a different set of user scenarios under that account. The timeout between the start of each new script was from 20 seconds to 10 minutes. This was done to emulate the most realistic behavior of a company’s employees who logged into the intranet portal and work inside it during the day.

Test results

An intranet Bitrix24 test portal (Enterprise edition) for 111,000 users installed on a cluster of 5 physical servers was able to provide smooth operation for 30,000 employees simultaneously. The maximum response time did not exceed 0,9 seconds in 95% of requests.

During the course of 1 hour, the aforementioned number of users generated:

• 39 news posts
• 4,614 messages in the Feed
• 4,685 comments
• 440 chats
• 3,386 instant messages
• 2,297 business processes
• 912 tasks
• 367 documents
• 92 workgroups and projects
• 291 events in Calendar
• 2,385 notifications

During the course of 24 hours, there will be generated:

• 110,736 new entries in the Feed
• 10,560 individual and group chats
• 2,208 workgroups and projects
• 112,440 comments
• 936 news posts
• 6,984 events in Calendar
• 21,888 tasks and assignments
• 8,808 documents
• 81,264 instant messages
• 57,240 notifications
• 55,128 business processes

Conclusion

The results of the load test of the self-hosted version of Bitrix24 (Enterprise edition) confirmed the high performance of the platform and stability in high load conditions. The demo portal deployed in a cluster solution of 5 servers ensured the simultaneous operation of 30 thousand users, which corresponds to the approximate load profile for a large corporation that employs 100-200 thousand employees.

At the same time, the system provided fast (even by the standards of Internet projects) response not exceeding 1 second for the majority of requests, which ensures a smooth user experience. The web cluster technology has once again proven its performance and reliability.

The testing methodology, scenarios, and load profiles as well as the use of physical user profiles allowed us to get the test conditions that were as close to real life as possible. This means that the results of this test can be used as a reference for designing new projects.

It’s also worth noting that our goal was not to maximize the amount of generated content on the portal during the test (news, tasks, documents, messages, and others). The goal was to create an accurate simulation of real-life user behavior. Still, the speed of adding information to the portal was also very high, which should be more than enough to satisfy the needs of even the most demanding clients.

We are ready to provide the testing methodology, portal configuration, and test environment upon request to our corporate customers and integrators to serve as a basis for conducting full-scale testing of custom-developed portals or their prototypes, testing hypotheses, calculating the safety factor of our solution under peak loads.

The test environment can be further customized - e.g., by adding integrations with a company’s corporate systems to ensure more accurate results.