tag:status.caes.ucdavis.edu,2005:/historyUC Davis: CA&ES Dean's Office Status - Incident History2024-03-28T17:06:04-07:00UC Davis: CA&ES Dean's Officetag:status.caes.ucdavis.edu,2005:Incident/195010982023-12-22T06:49:29-08:002023-12-22T06:49:29-08:00PrePurchasing and Aggie Enterprise<p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>06:49</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>13:32</var> PST</small><br><strong>Monitoring</strong> - It should be fixed now.<br />Double checking.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>12:42</var> PST</small><br><strong>Investigating</strong> - An issue with our permissions into Aggie Enterprise is causing order to get an error page.<br />We are aware of what the issue is and are working to get it fixed.</p>tag:status.caes.ucdavis.edu,2005:Incident/194806812023-12-19T19:50:42-08:002023-12-19T19:50:42-08:00PrePurchasing upgrade to Aggie Enterprise<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>19:50</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>19:49</var> PST</small><br><strong>Update</strong> - Update is complete. Currently some of the functionality on the order page to pick an external AE chart string isn't working. But this is working on the workgroup admin side of things.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>18:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>07:28</var> PST</small><br><strong>Scheduled</strong> - To allow Departmental Admins to start updating accounts and other Aggie Enterprise information before the go live date in January, we will be deploying the Aggie Enterprise changes.<br /><br />Functionally, this means that new requests will no longer route to account managers outside of the workgroup. Requests may continue to be created, approved, and completed within a workgroup before Aggie Enterprise goes live. New AE Chart Strings will need to be added to the workgroup or entered by users by January 4 to use the new requisition upload function when Aggie Enterprise goes live. <br /><br />PrePurchasing will be unavailable for part or all of this maintenance.</p>tag:status.caes.ucdavis.edu,2005:Incident/176492002023-06-21T20:38:17-07:002023-06-21T20:38:17-07:00UCDNet4 Network Maintenance<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>20:38</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>19:42</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>19:42</var> PDT</small><br><strong>Investigating</strong> - Please see: https://status.ucdavis.edu</p>tag:status.caes.ucdavis.edu,2005:Incident/175077682023-06-08T06:06:17-07:002023-06-08T06:06:17-07:00CA&ES Remote Desktop<p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>06:06</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>19:01</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.caes.ucdavis.edu,2005:Incident/172297232023-05-12T14:24:25-07:002023-05-12T14:24:25-07:00PrePurchasing Attachments<p><small>May <var data-var='date'>12</var>, <var data-var='time'>14:24</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>13:57</var> PDT</small><br><strong>Update</strong> - We continue to finetune the cloudflare firewall to be more permissive. <br />Still monitoring.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>09:07</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>08:45</var> PDT</small><br><strong>Monitoring</strong> - We had a cloudflare issue with uploading attachments. Fix put in place.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>08:37</var> PDT</small><br><strong>Investigating</strong> - There appears to be a problem with uploading attachments in PrePurchasing</p>tag:status.caes.ucdavis.edu,2005:Incident/115760292022-10-07T08:01:37-07:002022-10-07T08:01:37-07:00Data center generator/ATS maintenance<p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>08:01</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>18:31</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>18:23</var> PDT</small><br><strong>Scheduled</strong> - The campus data center is conducting a generator test early tomorrow morning.<br /> <br />I’ll need to shut down our servers hosted at the campus data center tonight at 6:00PM to prepare for the test. I will turn our servers back on tomorrow morning as soon as the test completes. Hopefully, everything will be back online by 8:00AM but I’ll send out a tech notice if the generator maintenance takes longer than expected. Nothing hosted at the Quest data center will be impacted by the maintenance tomorrow morning.<br /> <br />The following client services will be unavailable from 6PM tonight to 8AM tomorrow morning:<br /><br />CAES Remote Desktop<br />CAES VPN</p>tag:status.caes.ucdavis.edu,2005:Incident/108940672022-08-23T14:02:41-07:002022-08-23T14:02:41-07:00Update for PrePurchasing<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>14:02</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>13:31</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>13:19</var> PDT</small><br><strong>Scheduled</strong> - Update OPP https://computing.caes.ucdavis.edu/documentation/purchasing/Aggie-Enterprise-Updates</p>tag:status.caes.ucdavis.edu,2005:Incident/103818382022-06-22T20:24:58-07:002022-06-22T20:24:58-07:00Update for PrePurchasing<p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>20:24</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>13:01</var> PDT</small><br><strong>Scheduled</strong> - PrePurchasing May be unavailable for some or part of this time.</p>tag:status.caes.ucdavis.edu,2005:Incident/103732512022-06-21T20:00:40-07:002022-06-22T13:02:36-07:00Update for PrePurchasing<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>20:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>19:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>13:28</var> PDT</small><br><strong>Scheduled</strong> - Website may be unavailable while it is being updated.</p>tag:status.caes.ucdavis.edu,2005:Incident/85466382021-11-15T17:57:19-08:002021-11-15T17:57:19-08:00PrePurchasing Emails<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>17:57</var> PST</small><br><strong>Resolved</strong> - Emails are sending. For PrePurchasing, the ones that didn't send should start going out but they may not be sent in the order they were created.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>17:36</var> PST</small><br><strong>Monitoring</strong> - They should be sending now.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>17:34</var> PST</small><br><strong>Identified</strong> - We have a billing issue with emails. Looking into getting it resolved.</p>tag:status.caes.ucdavis.edu,2005:Incident/62873942021-02-17T08:37:57-08:002021-02-17T08:37:57-08:00Prepurchasing and KFS<p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>08:37</var> PST</small><br><strong>Resolved</strong> - If you have problems, please submit a help ticket.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>06:48</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>08:12</var> PST</small><br><strong>Investigating</strong> - There is a problem with completing orders for KFS<br /><br />When this happens, the only option is to either wait and try again, or manually add it to KFS and select this option:<br />https://www.screencast.com/t/DwatliyH<br /><br />This most likely related to this issue:<br />https://status.ucdavis.edu/incidents/dr92xblylg71</p>tag:status.caes.ucdavis.edu,2005:Incident/61938262021-02-09T16:00:31-08:002021-02-09T16:00:31-08:00Registration Upgrade to CyberSource<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>16:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>14:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>08:36</var> PST</small><br><strong>Scheduled</strong> - Registration will be unavailable during some or all of this time.</p>tag:status.caes.ucdavis.edu,2005:Incident/60809922021-01-21T19:30:35-08:002021-01-21T19:30:35-08:00Update PrePurchasing<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>19:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>19:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>13:34</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.caes.ucdavis.edu,2005:Incident/60729602021-01-20T16:18:27-08:002021-01-20T16:18:27-08:00An update didn't go smoothly<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>16:18</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>15:46</var> PST</small><br><strong>Monitoring</strong> - It should be running correctly, and a little faster now.</p>tag:status.caes.ucdavis.edu,2005:Incident/60095612021-01-13T16:18:30-08:002021-01-13T16:18:30-08:00PrePurchasing and KFS<p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>16:18</var> PST</small><br><strong>Resolved</strong> - This should be fixed now.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>15:39</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>15:39</var> PST</small><br><strong>Investigating</strong> - When this happens, the only option is to either wait and try again, or manually add it to KFS and select this option:<br />https://www.screencast.com/t/DwatliyH<br /><br />Just doubled checked with KFS, and they have an issue at the moment.<br /><br />=============<br />Yes, we use an AWS-DC Circuit which failed yesterday and has failed again today. Our Operations team is working with the NOC to figure out a redundant path. We are trying to see what our options are moving forward. <br />=============</p>tag:status.caes.ucdavis.edu,2005:Incident/56381782020-11-23T12:02:24-08:002020-11-23T12:02:24-08:00PrePurchasing KFS<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>12:02</var> PST</small><br><strong>Resolved</strong> - Services have been restored. The issue was caused by a physical fiber cut.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>10:20</var> PST</small><br><strong>Identified</strong> - KFS is having issues that is affecting PrePurchasing<br /><br />Our Kuali Financials endpoints are being affected by an AWS direct connect outage. We are working to reroute our endpoints and update all of our network rules. I’ll let you know when we are back up</p>tag:status.caes.ucdavis.edu,2005:Incident/46649882020-07-20T12:00:35-07:002020-07-20T12:00:36-07:00Network Maintenance<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:54</var> PDT</small><br><strong>Scheduled</strong> - The Computing Resources Unit will be performing some emergency maintenance of our primary firewall. The maintenance should take no longer than one hour. Please email us at support@caes.ucdavis.edu if you have any questions.</p>tag:status.caes.ucdavis.edu,2005:Incident/30776552019-10-07T21:31:03-07:002019-10-07T21:31:03-07:00PrePurchasing - Update indexes<p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>21:31</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>20:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>10:11</var> PDT</small><br><strong>Scheduled</strong> - There will be system downtime to update the application indexes tonight from 8pm-11pm. Sorry for any inconvenience.</p>tag:status.caes.ucdavis.edu,2005:Incident/23654322019-05-02T15:29:54-07:002019-05-02T15:29:54-07:00Azure Database Connectivity<p><small>May <var data-var='date'> 2</var>, <var data-var='time'>15:29</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>14:20</var> PDT</small><br><strong>Monitoring</strong> - It seems to be back up, we will keep an eye on it</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>13:31</var> PDT</small><br><strong>Investigating</strong> - There seems to be an Azure Database issue that is affecting several of our application</p>tag:status.caes.ucdavis.edu,2005:Incident/22495072019-03-08T09:31:59-08:002019-03-08T09:31:59-08:00Ace Slow<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>09:31</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>09:01</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>09:00</var> PST</small><br><strong>Monitoring</strong> - Increased app and database resources. Monitoring.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>08:50</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.caes.ucdavis.edu,2005:Incident/22201462019-02-21T13:35:45-08:002019-02-21T13:35:45-08:00Campus Networking Outage<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>13:35</var> PST</small><br><strong>Resolved</strong> - Campus has resolved the DNS routing issue and access to impacted resources have been restored.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>13:16</var> PST</small><br><strong>Identified</strong> - Campus is currently experiencing a networking outage, including DNS routing, that is impacting access to Give, ACE, PrePurchasing, Registration, and the CA&ES Remote Desktop service. Service will return to normal once the campus networking outage is resolved.<br /><br />For current details on the campus outage, please visit https://status.ucdavis.edu/incidents/8ltrs21yx4jt</p>tag:status.caes.ucdavis.edu,2005:Incident/20334682018-11-13T11:06:15-08:002018-11-13T11:06:15-08:00PrePurchasing slow/unavailable<p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>11:06</var> PST</small><br><strong>Resolved</strong> - Looks like everything is fine.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>11:05</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>09:47</var> PST</small><br><strong>Monitoring</strong> - It looks like it is working now. We will be monitoring it just in case.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>09:28</var> PST</small><br><strong>Investigating</strong> - This appears to be a campus issue, it was working fine from home this morning.</p>tag:status.caes.ucdavis.edu,2005:Incident/18624152018-08-16T07:14:16-07:002018-08-16T07:14:16-07:00PrePurcasing sending multiple emails<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>07:14</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:01</var> PDT</small><br><strong>Monitoring</strong> - We have deployed a fix and are monitoring. If you have been experiencing this issue, you should log out and log in again (preferably close your browser window)</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>13:22</var> PDT</small><br><strong>Update</strong> - Part of the issue seems to be if the user uses any uppercase chanters in their user name when logging into CAS</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>11:39</var> PDT</small><br><strong>Update</strong> - Fix didn't work correctly. Still need to investigate further.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>11:11</var> PDT</small><br><strong>Identified</strong> - Identified problem. We will be deploying a fix shortly.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>10:27</var> PDT</small><br><strong>Update</strong> - If you are receiving duplicate emails every 5 minutes, you may want to change your email settings to send daily until we fix this.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>10:16</var> PDT</small><br><strong>Investigating</strong> - PrePurchasing is sending multiple emails for some users.<br />We have identified the problem, but are investigating what is causing the problem.</p>tag:status.caes.ucdavis.edu,2005:Incident/16796272018-04-09T08:47:40-07:002018-04-09T08:47:40-07:00PrePurchasing<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>08:47</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>07:04</var> PDT</small><br><strong>Identified</strong> - Our Cert expired over the weekend, it should be fixed today (Monday).</p>tag:status.caes.ucdavis.edu,2005:Incident/16650612018-03-29T16:41:31-07:002018-03-29T16:41:32-07:00PrePurchasing KFS Security Token Error<p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>16:41</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>13:19</var> PDT</small><br><strong>Monitoring</strong> - KFS deployed their fix. If you have problems, please submit a help ticket.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>09:27</var> PDT</small><br><strong>Identified</strong> - KFS is aware of the problem and intend to do an emergency update at noon:
<br />
<br />KFS:
<br />Another issue occurred affecting the security token and we are planning an emergency update at the noon hour today. We will be sending out a message to the FIS-info listserv as soon as we get the green light.</p>