Upgrading to 9.4.0.206 Monitoring is falling behind schedule & Some monitors not getting to run

Since upgrading to 9.4.0.206 (upgraded from 9.3.0.77) we're started getting these error email alerts from all 14 of our PA File Sight satellites, which are on different physical hardware and at different physical locations:

Error: Monitoring is falling behind schedule (likely because of some other issue) There are over 3660 background tasks queued to run. This is unusually high and could lead to high memory usage or other problems. The service/Satellite log file will show what is in the queue. Search for "Background Thread Pool Log".

Error: Some monitors not getting to run Monitor I: includes XXX on XXX has been running for over 12h 11m. It might be locked. (MID 2, TID:06684, WaitRet:258)

We we're not getting any of these alerts and everything seemed to be working fine until we performed the upgrade last week and then upgraded the satellites when asked. When I check the performance on the satellites I barely see any CPU usage or disk usage, and the filesightsattelite.exe is typically at 0% CPU. There is little file activity on these servers, especially on a weekend.

The service/Satellite log file doesn't really show me anything I can understand; here is one of the satellite logs:

04-28-2024, 22:38:05.364, TID:05000, PID:04240, ERROR: Background Thread Pool Log TID 5808, Runtime: 74752 sec, AsyncProcReq - REQ:F91F1E2838B5B4833CF583D4C88F35DA42AF1B58F4CEE50979E58BF2D382840D RID_FS_GET_SOURCECOMP_ID rTID:03848 TID 1740, Runtime: 75296 sec, AsyncProcReq - REQ:AD045E1AFFB450417E9B68A70EF4C7572158F3A652540C0E7F2D5AC12D85E5CF RID_FS_GETCOMPID rTID:04752 TID 4900, Runtime: 75359 sec, AsyncProcReq - REQ:E9844A27E9EECFEF13F9240361AB5820DE90658E8DBFDEBB10173D5246E899BC RID_FS_GET_SOURCECOMP_ID rTID:03064 TID 4904, Runtime: 75359 sec, AsyncProcReq - REQ:121EE045CC66769830673707951594C71BA2EA4700ED6F987F7651CAA90C961A RID_FS_GET_SOURCECOMP_ID rTID:04068 TID 4908, Runtime: 75362 sec, AsyncProcReq - REQ:7275E0C58C0566169680CA81D58EA36E8EA1CDCE290B28924C427EC16E920F04 RID_FS_GET_SOURCECOMP_ID rTID:03084 TID 4912, Runtime: 75364 sec, AsyncProcReq - REQ:E7427D8B233AEE7BFD444C195A9D720282466C3A721C77AC0A43C495BF3DFD2B RID_FS_GET_SOURCECOMP_ID rTID:04564 TID 4916, Runtime: 75347 sec, AsyncProcReq - REQ:D7FD3B0DEE082C24DB36C0D1EB6E012E3E6F687CF16B21590E0CC000B9CC1C23 RID_FS_GETCOMPID rTID:04752 TID 4920, Runtime: 75363 sec, AsyncProcReq - REQ:48C97DD5E103A7F739AA64268AE44614C6051766996B985FC87C19514DBC2FED RID_FS_GETCOMPID rTID:00736 TID 4924, Runtime: 75405 sec, FSLEGACYDB TID 4928, Runtime: 75357 sec, AsyncProcReq - REQ:EF2A740F74F8F404C46A7E82F361FBCFDF17AB5502A3730DAD1DFE13B608709F RID_FS_GET_SOURCECOMP_ID rTID:00740 TID 4932, Runtime: 75349 sec, AsyncProcReq - REQ:2A51CEE694DBCCEC733701229298E75436329F86B25DDAEDD456FB6E097E549A RID_FS_GETCOMPID rTID:01696 TID 4936, Runtime: 75364 sec, AsyncProcReq - REQ:42883C542519D634657C29B2113205AF4A47BF16A0BD1231D3B8A8E0EE062771 RID_FS_GET_SOURCECOMP_ID rTID:03848 TID 4940, Runtime: 75348 sec, AsyncProcReq - REQ:E427EC96081447E1336DD37F3F95EA43F04FE4ABC6289C2061B3C877102EF62A RID_FS_GETCOMPID rTID:05984 TID 4944, Runtime: 75341 sec, AsyncProcReq - REQ:3DCBB749EF84E0F1B88E48982FE1F38AEB75AEA54C71BDC30C680187D019990E RID_FS_GETCOMPID rTID:00740 TID 4948, Runtime: 75346 sec, AsyncProcReq - REQ:FB8E6F76204A0AF9A06D0CD550B7BB0B07B6C68EBCFE7D4C9F973CF694116C74 RID_FS_GET_SOURCECOMP_ID rTID:05112 TID 4952, Runtime: 75363 sec, AsyncProcReq - REQ:9A366C822B10B51D54390FB80BCEFA7158D236A023C560D973BDCB5DA9E003CF RID_FS_GETCOMPID rTID:03044 TID 2108, Runtime: 74758 sec, AsyncProcReq - REQ:DA754A66219A999D6C2A1332DDA44511C59D32216C8F9907C3AA53CC3AAB2460 RID_FS_GET_SOURCECOMP_ID rTID:01056 TID 3976, Runtime: 74750 sec, AsyncProcReq - REQ:E7BBC93FC027FA4B600B3CA4798BD70CE1BBB8EBB1F151A46BB127D5B6D4190D RID_FS_GETCOMPID rTID:04136 TID 3344, Runtime: 10803 sec, MsgDisp TID 3344, Runtime: 10803 sec, MsgDisp TID 4968, Runtime: 75360 sec, AsyncProcReq - REQ:6BA83AE6A180DA23B283A36E499EE819BF334DA8D349C72F31BAC4B86ABAF013 RID_FS_GETCOMPID rTID:01056 TID 4976, Runtime: 75344 sec, AsyncProcReq - REQ:1B967AED4F2BF3CCE479323F0EA7A7B73C48D0F78B58F2F9747C56E386CEA4C4 RID_FS_GET_SOURCECOMP_ID rTID:02172 TID 4980, Runtime: 75359 sec, AsyncProcReq - REQ:0AE626EDBABC5E49B2133D552B2BF91BC966F0F2C11B24BD3B4F2B060A734135 RID_FS_GET_SOURCECOMP_ID rTID:02780 TID 4988, Runtime: 75361 sec, AsyncProcReq - REQ:4B419609B58EB8357504FA6A6773D4B4CABD1E1FB2592358DF1DBDC9A7A5B59A RID_FS_GET_SOURCECOMP_ID rTID:05688 TID 4992, Runtime: 75316 sec, AsyncProcReq - REQ:B23BCFC9B666C77DD7086942BA4E8818F56D771111D7D591E39D423424D9F4CD RID_FS_GETCOMPID rTID:04068 TID 4996, Runtime: 75363 sec, AsyncProcReq - REQ:9465FCD3E8985CF79697A1CBBE856DDA042B0ED4909CABC02E78BD427B0B3D60 RID_FS_GET_SOURCECOMP_ID rTID:01252 TID 5108, Runtime: 74757 sec, AsyncProcReq - REQ:D566D1FEEDA1FCF48E17437B8A5E6DF76790D938D36B0C756A456725C59D70C6 RID_FS_GET_SOURCECOMP_ID rTID:01388 TID 420, Runtime: 74755 sec, AsyncProcReq - REQ:23E10A47EA3EF595839343D80787CA85567903577C6686BE6087D742621B7CE6 RID_FS_GET_SOURCECOMP_ID rTID:03964 TID 3476, Runtime: 75130 sec, AsyncProcReq - REQ:865914A874802C838C30B94FF519D9A1CBA388279245E76C7EFE2573BB09E0D4 RID_FS_GET_SOURCECOMP_ID rTID:03964 TID 3244, Runtime: 74141 sec, AsyncProcReq - REQ:0E4B6917B42EAF371B55196E41117082F725D94C9D1B421E15175794B31C18AB RID_FS_GETCOMPID rTID:04000 TID 2512, Runtime: 74705 sec, AsyncProcReq - REQ:D8807136DF136DC7EC99E160A1CBCC38188CCEAAD67BC2E87BDE03F118502EBF RID_FS_GETCOMPID rTID:02964 TID 4372, Runtime: 75347 sec, AsyncProcReq - REQ:7A20C23AF8249F6DA8AD1FC4F8A1312C45A670A51837DBF2021D9AF05E656509 RID_FS_GETCOMPID rTID:01304 TID 2424, Runtime: 74738 sec, AsyncProcReq - REQ:4A2E6F7A86B339D7B43A3F0BAF92556D2497CD2FEB0CB5A1EF07DBCB3FB4E5EF RID_FS_GETCOMPID rTID:01020 TID 1084, Runtime: 75348 sec, AsyncProcReq - REQ:815CD1F743089FA1015E820A364A46D74B0A0EC462918579AE961703DBF8E072 RID_FS_GETCOMPID rTID:04656 TID 4408, Runtime: 75347 sec, AsyncProcReq - REQ:39390AD0DEFE3424BC1FA194EE3E3693E4C0B9E9DAE733119BD875F17FF67935 RID_FS_GET_SOURCECOMP_ID rTID:02496 TID 1940, Runtime: 74757 sec, AsyncProcReq - REQ:2B93AFA66EFA35485C637641EBCB8818BA3DB0739828A91703A845549A7BE570 RID_FS_GET_SOURCECOMP_ID rTID:02056 TID 5756, Runtime: 74143 sec, AsyncProcReq - REQ:C20F77AC69E581DE43A81E2657032E0806532E512A77053E0DA72C0DE36D3F6A RID_FS_GET_SOURCECOMP_ID rTID:00560 TID 3660, Runtime: 74722 sec, AsyncProcReq - REQ:EF98C68582AB9C84A33A1E956BF4E2C0A4CA1507DAB5A8E55231E746251A0EEC RID_FS_GET_SOURCECOMP_ID rTID:03964 TID 4512, Runtime: 75328 sec, AsyncProcReq - REQ:8E55EEA02CEC422645EB993582AD5558552BE9EA709C727E4F4F3FEB1432EC66 RID_FS_GET_SOURCECOMP_ID rTID:02780 TID 4760, Runtime: 75130 sec, AsyncProcReq - REQ:1AEAF6CB10CFD72C092F7E54BD789A25608A918CF86CFEE5BC514A939DB8BDF1 RID_FS_GETCOMPID rTID:04136 TID 4108, Runtime: 75315 sec, AsyncProcReq - REQ:BE1D4B8454478EF1FDE3A85BF3BCA608333AA2300190069DBB992DC01A56400F RID_FS_GETCOMPID rTID:04656 TID 3500, Runtime: 74504 sec, FSEndPHistUpd TID 3500, Runtime: 74504 sec, FSEndPHistUpd TID 688, Runtime: 13 sec, BCast-RID_SEND_FS_BLOCKLIST TID 472, Runtime: 75283 sec, FSLIg TID 472, Runtime: 75283 sec, FSLIg TID 5128, Runtime: 74753 sec, AsyncProcReq - REQ:62275E39C0BC10DF77C970C85F5D23E158399066E7716E952416027F894D0FAA RID_FS_GET_SOURCECOMP_ID rTID:05984 TID 2180, Runtime: 74742 sec, AsyncProcReq - REQ:9BC4096BB318FC4B5E52E12E533A5CC4F922EF1224AC1639777B55A5293CB727 RID_FS_GET_SOURCECOMP_ID rTID:03064 TID 1884, Runtime: 74754 sec, AsyncProcReq - REQ:64F8BF5317331A5EE0F88DC09C65BE308105D5847A4368B75F438B2E3D405212 RID_FS_GET_SOURCECOMP_ID rTID:02536 TID 5744, Runtime: 74753 sec, AsyncProcReq - REQ:8751BB5DD0D43123BE74F0A544E48C0BAC0DD6C46CF8522757300C71DD1E3EFA RID_FS_GET_SOURCECOMP_ID rTID:04564 TID 1692, Runtime: 74752 sec, AsyncProcReq - REQ:65285C68C623F6CC87E77A86B736FE1D72370E793D1558C812EA6EF87271795C RID_FS_GETCOMPID rTID:05020 TID 2408, Runtime: 74140 sec, AsyncProcReq - REQ:4AF16F4A66AE9DCF9BF95E94E84A8528831A76957AE21B52C1AF473760C75BA0 RID_FS_GETCOMPID rTID:03848 TID 4040, Runtime: 75128 sec, AsyncProcReq - REQ:07D084BEC849A5DEEDC2BD42870E6B6CD74F240F5CA2176EA6C37BBA6A213E77 RID_FS_GETCOMPID rTID:03788 TID 2584, Runtime: 74737 sec, AsyncProcReq - REQ:1E37E049FD59EF33969BC3C8BFECB6759CFEAEECAEADE2E3DC5D6F8B9D7F68FB RID_FS_GETCOMPID rTID:02896 TID 5112, Runtime: 74609 sec, AsyncProcReq - REQ:3113C5CA15AFA7FD90BFB1D36BB0646E65E421BFAB03C8FAB4717EE2DDC0099C RID_FS_GETCOMPID rTID:04000 TID 2820, Runtime: 74520 sec, AsyncProcReq - REQ:F2F9A37223AE8E601DF49B000E3A6B278A52ED4FC61172ACD11ED0527ECF56FC RID_FS_GETCOMPID rTID:05812 TID 5728, Runtime: 74740 sec, AsyncProcReq - REQ:9500304764D6368FAF0F760E66899497C8184C8C56C8663EDB5D150B8AEF42E0 RID_FS_GET_SOURCECOMP_ID rTID:05720 TID 5632, Runtime: 74752 sec, AsyncProcReq - REQ:432B522DD978866E211FE6AA5EA3E921177C70A649AEE76DB96027B8201756B9 RID_FS_GETCOMPID rTID:05116 TID 3336, Runtime: 74143 sec, AsyncProcReq - REQ:40C19D019CDE9F663BF1090040D1394883F2B16A4B9538EBEC010702EE14B416 RID_FS_GET_SOURCECOMP_ID rTID:04752 TID 652, Runtime: 71801 sec, FSDelOld TID 652, Runtime: 71801 sec, FSDelOld TID 2868, Runtime: 74739 sec, AsyncProcReq - REQ:2CFE490FE5AA07FBEC4A465AC64203C967EC57E50A212B91AC32F361EF22DB02 RID_FS_GET_SOURCECOMP_ID rTID:05664 TID 5308, Runtime: 74737 sec, AsyncProcReq - REQ:E566204548A584DF7A29AC04266FEE24F3F4F2D955C39ACF905881EFF9425CA2 RID_FS_GETCOMPID rTID:02172 TID 2040, Runtime: 74685 sec, AsyncProcReq - REQ:7E306EBC9A505FF815D5487BA228DA597CA8F0C009662D31C7DB5BE6B2E51BB6 RID_FS_GETCOMPID rTID:03964 TID 1868, Runtime: 74758 sec, AsyncProcReq - REQ:A93F89136F5B2FFF2A7762F54FEDC2CCEEB1E1D25A5F04A34064269E9B3DB57F RID_FS_GET_SOURCECOMP_ID rTID:02896 TID 1100, Runtime: 74142 sec, AsyncProcReq - REQ:4C31082923FB225AD320E07E1BBB759B664BA432D6AB81568ED505B5546E9539 RID_FS_GET_SOURCECOMP_ID rTID:01304 TID 5604, Runtime: 74135 sec, AsyncProcReq - REQ:C6DBE9B1B43D9EB0096872D68074BF9EF49A9E91B806D1A265BC368836713759 RID_FS_GET_SOURCECOMP_ID rTID:01008 TID 3276, Runtime: 74735 sec, AsyncProcReq - REQ:9BFDA1C52647A3CAB7FE655FD3B4DE72B14AEBE6C916B025EA9AA7CBAC280C5A RID_FS_GETCOMPID rTID:04068 TID 3928, Runtime: 74142 sec, AsyncProcReq - REQ:7C0E4D0B354478DD44DA3319BF303CD334E69B903CB2E000CF2F5B62C808BCEE RID_FS_GET_SOURCECOMP_ID rTID:00736 TID 488, Runtime: 74146 sec, AsyncProcReq - REQ:A84E2E67D16A42744A8682B2E37C514EEC4ACDDC79330A7153A5E5E8777D2CB2 RID_FS_GET_SOURCECOMP_ID rTID:05720 TID 2624, Runtime: 74737 sec, AsyncProcReq - REQ:43EF40964F64A6D98972A181165B6C11727ACF65EF36A573F75FF3F7D7AD5837 RID_FS_GETCOMPID rTID:00564 TID 3320, Runtime: 74731 sec, AsyncProcReq - REQ:DD99169DCB2A898CB2E96E461258A71AB7E0A1B79EE9BEED015EE840BF857EFE RID_FS_GETCOMPID rTID:03680 TID 2828, Runtime: 74142 sec, AsyncProcReq - REQ:7D71D4E6B7079AA26F8440E51E4AD0E3F60B838803FD0030B283FF40C4D7D28D RID_FS_GETCOMPID rTID:02496 TID 1492, Runtime: 74704 sec, AsyncProcReq - REQ:24B4C0DD195B793AB775D4EF7D2922B174B633483E3F2CEDF1FE33F45FF09E98 RID_FS_GETCOMPID rTID:01484 TID 6092, Runtime: 74147 sec, AsyncProcReq - REQ:BB2244E7BCDE1A2782F6A3571920BAD7A0C4FB3F2D15CE0A95FB0019A9840B61 RID_FS_GET_SOURCECOMP_ID rTID:00820 TID 1572, Runtime: 74525 sec, AsyncProcReq - REQ:5793E30B2D69DDAE0F3088190CAD9F24536ADE9C92800D014AA1EEF38AF8D066 RID_FS_GET_SOURCECOMP_ID rTID:03964 TID 3080, Runtime: 74517 sec, AsyncProcReq - REQ:94D5C0C61F2C0925F0C353760A12223D95FCC7CEA8E201DF50AA63B2934491A1 RID_FS_GETCOMPID rTID:03848 TID 5436, Runtime: 74148 sec, AsyncProcReq - REQ:74758772181537B83B334C7986F0526A5994EB0AEEB99341EE6172C9F653D389 RID_FS_GET_SOURCECOMP_ID rTID:01700 TID 5636, Runtime: 74274 sec, AsyncProcReq - REQ:B96E55C30BFD0CBAC1AC199F51A2161D0EE2A3C579D5FE4122A2E0C7B34C79A4 RID_FS_GETCOMPID rTID:02764 TID 984, Runtime: 74145 sec, AsyncProcReq - REQ:C2F71CA8008AEF4E111A5E4DBCE3A656EC51C623B19437E0F713F44AD1B93CDC RID_FS_GET_SOURCECOMP_ID rTID:02764 TID 432, Runtime: 74146 sec, AsyncProcReq - REQ:9B8FAA4BEA946F40D9EAB7AE2AC015056F66C9D6AE8314B379F857AF75B49366 RID_FS_GET_SOURCECOMP_ID rTID:04564 TID 2548, Runtime: 74136 sec, AsyncProcReq - REQ:7E0A1CAB19829C2924E237570191A45A2EFD16B0CFE6325724AA8B4BF283507A RID_FS_GET_SOURCECOMP_ID rTID:02032 04-28-2024, 22:38:05.364, TID:05000, PID:04240, ERROR: 3491 AsyncProcReq - REQ:B02737F0D519F74FFC1313DF92AB2612A786E047B6ACE877D5C96B7DD586C965 RID_FS_GET_SOURCECOMP_ID rTID:04368(multi) 04-28-2024, 22:38:05.380, TID:03804, PID:04240, ThreadProc -> FSEndPLicd rTID:05000 04-28-2024, 22:38:05.914, TID:02988, PID:04240, ThreadProc -> MaintChk rTID:05000 04-28-2024, 22:38:06.041, TID:05316, PID:04240, SVC: Got list of monitors to watch readyToRun: 0 04-28-2024, 22:38:06.418, TID:01876, PID:04240, ThreadProc -> CleanFields rTID:05000 04-28-2024, 22:38:07.935, TID:02988, PID:04240, ThreadProc -> CFSA rTID:05000 04-28-2024, 22:38:08.548, TID:03568, PID:04240, EMAIL_SEND: Trying first SMTP server mail.XXXX.yy 04-28-2024, 22:38:08.596, TID:03568, PID:04240, EMAIL_SEND: Successfully sent mail to XXXX@XXXXXXXX.com via mail.XXXX.yy [Alert from DLP : Error: Monitoring is falling behind schedule (likely because of some other issue)], caller=EmailAction 04-28-2024, 22:38:08.596, TID:03568, PID:04240, AUDIT_LOG: Evt=Sent Email User=XXXX@XXXXXXXX.com Source=DLP Ver=9.4.0.206 GID 4294967295 CID 0 MID 0 Desc=Subject: Alert from DLP : Error: Monitoring is falling behind schedule (likely because of some other issue)