WebSVN loads index page with lots of repos very slowly.

[JimmieJimmie]

We are using WebSVN 2.5 with Apache 2.4.37 and have about 120 Repositories and authorization via active directory. WebSVN is very slow! It takes about 50 seconds to load the index page with all repositories (even on the server machine using localhost/websvn)! According to Chrome the TTFB is about 40 second. So what can be the reason for this long response time?
The standard svn web page is loaded in about 500 ms.

[ams-tschoening]

Guten Tag JimmieJimmie, am Dienstag, 23. April 2019 um 17:28 schrieben Sie:

We are using WebSVN 2.5 with Apache 2.4.37 and have about 120 Repositories and authorization via active directory.

Authorization or authentication? While that might sound nitpicking, I wonder if WebSVN supports authorization using Active Directory at all and am somewhat sure it doesn't. Instead, authentication is handled by httpd using Active Directory, but authorization should be handled by some shell tool called "svnauthz" using paths to and within the repos. http://doc.mawan.de/subversion/svn_1.8_releasenotes.html#svnauthz_accessof

include/authz.php

It might make sense to provide the config regarding the paths to your SVN-repos and your access files, if at all. Without access files no authorization should happen at all and therefore "svnauthz" should not be called. If I was you I would use Process Monitor to see what happens during a request. https://docs.microsoft.com/en-us/sysinternals/downloads/procmon

The standard svn web page is loaded in about 500 ms.

You mean mod_dav_svn? Mit freundlichen Grüßen, Thorsten Schöning

…

-- Thorsten Schöning E-Mail: Thorsten.Schoening@AM-SoFT.de AM-SoFT IT-Systeme http://www.AM-SoFT.de/ Telefon...........05151- 9468- 55 Fax...............05151- 9468- 88 Mobil..............0178-8 9468- 04 AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow

[michael-o]

Cannot confirm. Works lightning fast with mod_auth_gssapi and svnauthz here. Provide more information. @

[JimmieJimmie]

Svn and WebSvn use the same authentication via Apache and Active Directory and the same access file for the subversion repository permissions (authz). Svn works fine (mod_dav_svn), WebSvn is really slow.
For the authentication in Apache the modules mod_authnz_sspi and mod_authz_svn are used.

WebSVN configuration in Apache:
` <Location /websvn>
AuthName "WebSVN Repository Login"
AuthType SSPI
SSPIAuth On
SSPIAuthoritative On
SSPIDomain DomainName

SSPIOmitDomain Off

SSPIOfferBasic On
SSPIPerRequestAuth Off 	
SSPIOfferSSPI On   
SSPIBasicPreferred Off

SSPIUsernameCase lower    
Require valid-sspi-user

`

The dialog for entering username and password occurs immediately, than 40 seconds nothing happens and the start / index page of WebSvn loads. What can be taking so long? Are there some additional WebSvn logs?

config.php.txt

[michael-o]

The config doesn't help. Unless you provide some debug logs we cannot help. It is likely an issue with your server, PHP module or something else. Reduce everything to a minimum and retry.

[JimmieJimmie]

It's a clean installation of Wampserver 3.1.7 (64 bit) on a new machine (Windows 10 64 bit, clean installation). Apache serves only Svn and WebSvn. MySQL and Maria DB in Wamp disabled.

Attached part of the access.log file, that contains only one call of "http://localhost/websvn" directly on the server machine with AccessFile disabled. It takes about 18 seconds to process the WebSVN GET request.
Calling "http://localhost/svn" takes 4 ms.

access_log_single_websvn.txt

[ams-tschoening]

Guten Tag JimmieJimmie, am Mittwoch, 24. April 2019 um 07:17 schrieben Sie:

What can be taking so long?

As said before, start with changing your config to not use an access file and additionally use Process Monitor to see what happens during requests. With an access file svnauthz is called and needs to be available somewhere, so it might get looked up in your PATH and whatever, which might take some time. Do you even have that installed? I'm not even sure where to get it from for Windows, it's e.g. not part of TortoiseSVN by default. An additional source for waiting time might be your configured "parentPath", because WebSVN is iterating that for all folders checking things to guess which of those are SVN-repos.

configclass.php::ParentPath::findRepository

There e.g. might be some additional, unnecessary deep iteration because of some reason. Some of my own software had similar problems in the past when used with SVN working copies, because it started iterating ".svn"-dirs for no reason, which impacted performance a lot. mod_dav_svn might simply work differently and if you additionally consider AV-software most likely available on your Windows maybe scanning things for WebSVN it doesn't do for mod_dav_svn, some additional milliseconds of overhead easily add up on each other. All those things would be visible by Process Monitor, that's why I suggest it always in those cases. You simply need to debug this further and the things I mentioned are the first I would have a look at. Waiting time most likely comes from either network, I/O or CPU somewhere and you need to find the bottleneck.

Are there some additional WebSvn logs?

Sadly not, there's only what you see in the webserver logs or what PHP provides independent of any application. Process Monitor is your best choice currently in my opinion, else you need to add debugging statements to WebSVN. Mit freundlichen Grüßen, Thorsten Schöning

…

-- Thorsten Schöning E-Mail: Thorsten.Schoening@AM-SoFT.de AM-SoFT IT-Systeme http://www.AM-SoFT.de/ Telefon...........05151- 9468- 55 Fax...............05151- 9468- 88 Mobil..............0178-8 9468- 04 AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow

[michael-o]

20 seconds of process time seems very very long. Is this due to WAMP? I have never used this stack on Windows and hopefully never will.

[ams-tschoening]

Guten Tag JimmieJimmie, am Mittwoch, 24. April 2019 um 09:19 schrieben Sie:

Attached part of the access.log file, that contains only one call of "http://localhost/websvn" directly on the server machine with AccessFile disabled. It takes about 18 seconds to process the WebSVN GET request.

Which is obviously far less than before, so Process Monitor might tell you where the difference between with/without access file comes from. And at the same time it will tell you what happens during searching for repos to show up. You can test performance between using parentPath and manually adding all repos in the config as well. Shouldn't bee too difficult even for 120 repos, just print their names on the shell and copy&paste and stuff in your editor. Mit freundlichen Grüßen, Thorsten Schöning

…

-- Thorsten Schöning E-Mail: Thorsten.Schoening@AM-SoFT.de AM-SoFT IT-Systeme http://www.AM-SoFT.de/ Telefon...........05151- 9468- 55 Fax...............05151- 9468- 88 Mobil..............0178-8 9468- 04 AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow

[JimmieJimmie]

Yesterday, before opening this issue, I've already tested to add all of the repositories using $config->addRepository('NameToDisplay', 'file:///c:/Repository Path'); instead of using ParentPath, same slow behaviour.
Now I will try the ProcessMonitor and will report later.

[michael-o]

You might want to disable authc and authz and see if this makes a difference. One needs to narrow this down.

[JimmieJimmie]

WebSVN calls "svnauthz" for each repository. According to the ProcessMonitor this takes about 200 ms for single repository. We have 120 repositories, so 120 * 200 ms = 24 seconds.
In the ProcessMonitor I can see, that it takes about 100-120 ms from the command call from httpd.exe (cmd.exe /c svnauthz accessof -repository -path -username) till the real execution of svnauthz.exe.
Is there any way to speed up these calls?

[michael-o]

That's good news. svnauthz operates on per-repo basis and there is no interface to that what svnauthz does beside in C otherwise one could read this file in memory and reuse it. I don't see any at the moment because reimplementing the entire logic in PHP I would highly discourage.

I just checked the code of mod_*_svn. If I properly understand the stuff, the file is read and then all operations are performed.

Subversion also provides a listing of all repos, how long does this take? Should be way faster.

[michael-o]

At best, you'd raise this question with users@svn.apache.org. @brainy any smart idea how we can solve this better from within PHP?

[ams-tschoening]

WebSVN calls "svnauthz" for each repository.

Where did you get the binary from? Just in case I would like to test
myself.

According to the
ProcessMonitor this takes about 200 ms for single repository.

This is what I get on UB 16.04 LTS for an example call:

time svnauthz accessof %f/conf/authz --repository=%f --path=/trunk --username=[...]
[...]
real    0m0.009s
user    0m0.003s
sys     0m0.000s

The same for Git-Bash under Windows 10 with most likely disabled Defender:

real    0m0,055s
user    0m0,000s
sys     0m0,015s

Execution in Powershell is of some interest, as it jumps between 26 and ~160 ms:

PS C:\Program Files\CollabNet\Subversion Client> Measure-Command {.\svnauthz accessof "C:\[...]\authz" --repository=[...] --path=/trunk --username=[...]}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 26
Ticks             : 260325
TotalDays         : 3,01302083333333E-07
TotalHours        : 7,23125E-06
TotalMinutes      : 0,000433875
TotalSeconds      : 0,0260325
TotalMilliseconds : 26,0325

PS C:\Program Files\CollabNet\Subversion Client> Measure-Command {.\svnauthz accessof "C:\[...]\authz" --repository=[...] --path=/trunk --username=[...]}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 156
Ticks             : 1562773
TotalDays         : 1,8087650462963E-06
TotalHours        : 4,34103611111111E-05
TotalMinutes      : 0,00260462166666667
TotalSeconds      : 0,1562773
TotalMilliseconds : 156,2773

In the ProcessMonitor I can see, that it takes about 100-120 ms
from the command call from httpd.exe (cmd.exe /c svnauthz accessof
-repository -path -username) till the real execution of svnauthz.exe.
Is there any way to speed this calls up?

Might depend on what causes the delay, check AV-software, network logs
of Process Monitor, Thread Creation etc., not only I/O operations. I
doubt that reading the file itself into memory is slow, I guess it's
either parsing it or process creation itself already. You might want
to consider providing the Process Monitor logs to us as well, so that
others can have a look.

It doesn't seem like the command can be invoked with multiple
different repos and paths once or such, so we need to focus on
improving each invocation. Besides being slow, I'm with @michael-o
that WebSVN shouldn't reimplement that command (again).

[JimmieJimmie]

I'm using the newest version ot the Collab Subversion. Running svnauthz manually in the command line doesn't take long. It takes long, when the PHP code of WebSVN tries to run the svnauthz via the exec function. Apache/httpd needs some time (100-120 ms) from the call of the exec function till the cmd.exe is invoked. I don't know why it takes so long and how can I speed this up.