Completed Maintenance Tasks

Owned by Rami Jalal
Last updated: Dec 04, 2023

  • Oct 24, 2023 BANFAID Clone → Completed

    • We are planning to clone BANPPRD on 10/17 to synchronize it with the production environment following the Q2 upgrades this weekend. To ensure a smooth transition, please ensure that you have backed up all non-production work so that it can be reapplied after the cloning process. We anticipate that PPRD will be fully operational on 10/18 for regular use.

  • Oct 20, 2023 Admin Pages Maintenance 17:00 - 16:00 → Completed

    • An issue was discovered in the SWASINS Admin Page due to Q2 updates. A fix required a rebuild of the admin pages. While we are awaiting official release tags from Ellucian for the build, an experimental build has been deployed in BANTEST as a temporary solution.

    • We got enough testing approvals. Thank you all.

    • We are going to do the Admin Pages deployment today 10/20 at 17:00, and we expect an hour outage.

    • Impacted applications: Application Navigator/ Admin Pages/ BCM/ BAM/ Nelnet

  • Oct 10, 2023 BANPPRD Clone

    • We are planning to clone BANPPRD on 10/10 to synchronize it with the production environment following the Q2 upgrades this weekend. To ensure a smooth transition, please ensure that you have backed up all non-production work so that it can be reapplied after the cloning process. We anticipate that PPRD will be fully operational on 10/11 for regular use.

  • Oct 7, 2023 Banner Q2 Upgrades

    • Banner 2023 Q2 Production upgrades are scheduled for deployment on 10/07. To facilitate this, any Banner-related Appman jobs that remain incomplete will be halted at 08:00. Subsequently, all Banner-related applications will be temporarily unavailable during the deployment. The deployment process is set to begin promptly at 8:00 AM, with an expected completion time of 17:45. Following this, the environment will be open for testing, with stakeholder verification expected to conclude by 19:00.

    • Impacted applications:

      • Banner Admin Pages/ Application Navigator/ BCM/ BDM/ BAM/ Registration SSB/ Student SSB/ Faculty SSB/ Student API/ Integration API/ Banner Extensibility/ uAchive CSU and CSUP/ Banner 8

  • Oct 1, 2023

  • Sep 23, 2023 Quarterly Exadata Infrastructure Maintenance 09:00 - 16:00

    • Oracle will be performing maintenance on the Exadata Cloud at Customer hardware. There will be rolling restarts during this maintenance so while the database will remain available, any applications that are not compatible with rolling restarts will need to be shut down including AppMan. → BCM will be restarted by the end of this process. Student Registration (Prerequisites checks) might be affected.

  • Sep 3, 2023 BDM Restart on 9/3

    • To apply larger document size change.

    • Would need to hold any jobs or processes that insert documents to BDM.

    • No other systems would be down.

  • Aug 23, 2023 Non-prod DB upgrades:

    • The SIS team will conduct a scheduled shutdown of all Banner Applications in non-production environments from 8:00 AM to 12:00 PM. If the DB upgrade completes ahead of schedule, we will promptly restore the services and notify you accordingly.

  • Aug 14, 2023 Banner SSB outage - Now - Restarting all SSB services between 9:20 and 9:40 AM.

  • Aug 13, 2023 OS Patches - All Banner Applications are impacted (Admin Pages / BCM/ BAM / SSBs/ Banner 8 ) and uAchieve Fort Collins. Downtime during maintenance period 7:00AM - 01:00 PM.

  • Aug 3, 2023 Palo Alto Upgrade (Network) → Completed

    • During Summer Grades Freeze period

    • Failover – so no downtime, however connected users will require to reconnect to different servers during this period.

    • Potential issue for faculty grade entry as they will lose grades entered that are not saved yet. Put messages up on ARIESweb and PAWS to not enter grades between 9:45pm and midnight.

  • Jul 30, 2023 BDMS PROD Migration and Upgrade

    • bdmsprod.colostate.edu expected maintenance window (assuming no problems) is: 2023-07-30 from 08:00 - 17:00. Scot can send notification when we are complete and verified (if needed/desired). Desktop services will be deploying new versions of the BDM desktop application to users starting Monday 2023-07-31.

  • Jul 27, 2023 Banner Admin / BCM / BAM behind GlobalProtect even for on-campus users - for more details here is the source article.

    • BANTEST and BANUPGD are behind the GlobalProtect.

    • As of this morning, all Banner Administrative Applications (Admin Pages/BCM/BAM) are now behind GlobalProtect. To access these applications, you must connect to the GlobalProtect client, whether you are on or off-campus.
      This change has already been applied to BANTEST and BANDEVL environments, and it will be extended to other non-production environments soon.
      Please feel free to reach out if you have any questions or concerns.

    • What is changing?

      • Customers will need to log into Global Protect before accessing information in Oracle HR, Banner administrative pages, Banner Communication Management, and Banner Access Management

      • The GlobalProtect login will be required to access these applications from both on- and off-campus

      • Beginning July 27, the GlobalProtect login will change from prompting for the user’s NetID and instead require it in the format NetID@colostate.edu

      • Duo will need to be installed on customers devices if they have not already done so

  • Jul 21, 2023 - Jul 24, 2023 NetID Go-Live

  • Jul 8, 2023 Banner 2023 Q1 Upgrades - Saturday 8:00 - 15:00 → Completed

  • Jun 29, 2023 BANPROD - Banner Admin Servers restart Today at 17:00

    • A fix has been implemented to fix an issue with Transcript. In order for this fix to take an effect we need to restart Admin Servers . We are restarting Admin Servers today 17:00, the process might take up to 15 minutes.

  • Jun 21, 2023 BANPROD - MFA / DUO Factor Authentication for Admin Pages - Completed

    • We got all the approvals to implement DUO for Admin Pages / BCM / and BAM Thursday at 07:30 AM. We expect the implementation to take 10 min, and no downtime is expected.

  • For CSUP Campus Only: For Faculty and Staff that use SSB along with the Banner Administrative pages and Banner Communication Management (BCM), there is an issue with accessing both SSB and either the Admin pages or BCM simultaneously. For example, if you are logged into the Admin pages or BCM and open a new browser tab to log into SSB, you will get an error and won’t be able to log in. We’ve identified a solution for this: we need to apply MFA/DUO to the Administrative pages and BCM as well. We are anticipating that change will take place next week (the week of June 12). In the meantime, there is a work-around: If you are already using the Admin pages or BCM, open an incognito or private window to log in to SSB. → Resolved by applying DUO to admin applications Jun 21, 2023

  • Jun 6, 2023PROD BDM (Banner Document Management) Maintenance Today 12:00 PM → Completed

    • An issue was introduced to BDM after the latest Chrome updates. We need to shut down BDM for 15 min to resolve the issue and implement the fix. No other Banner related apps should be affected.

    • We will start BDM maintenance today at noon and expect the maintenance to end around 12:15 PM

  • Jun 3, 2023 - BANPROD Maintenance on Saturday 06/03 - 0800 - 1500 → Completed

    • The following will be implemented in BANPROD Saturday, 06/03. We are starting at 0800 and the expected finish time is 1500.

      • Exadata Upgrades

      • DUO Factor Authentication on All Banner SSBs

      • BCM Config changes

    • Exadata Upgrades are expected to consume most of the time. Meanwhile, we will apply the DUO factor on all Banner SSbs and the BCM configuration changes.
      Banner SSBs and BCM might be available earlier than the finish time, and we will make them available as soon as they are ready.

    • Appman / Jobsub / and some jobs that Student Registration depends on will be down during maintenance.

  • May 23, 2023 Banner Apps connectivity to DB is failing - Resolved

    • BANPROD, was overloaded with queries. This was causing issues with Banner and applications that use the Banner database. The app recycles cleared a lot of inactive sessions out of the DB, and it's back to reasonable levels again.

  • May 19, 2023 - All Banner Applications are failing on the authentication level. We are investigating. - Resolved:

    • The issue was EEID cluster DB accounts password expired which led to SSO not working for any of the SIS applications

      The network issue that resulted from the flood on the 6th floor affected our system access to debug the issue

  • May 8, 2023 BANPROD Admin Pages SSO issue - Completed

    • Users reported issues while signing into Admin Pages. It looks like one of the servers need to be fixed. That server has been disabled and out of the balance for now. We are working on fixing it and will add it back when it is ready. We are sorry for any inconvenience and I will keep you posted.

  • May 7, 2023 BANPROD - OS Patches + Banner Upgrades - Sunday 07:00 - 1500 → Completed

    • The following are planned to happen this Sunday starting at 7:00 and expecting to finish at 15:00:

      • OS Patches on all servers (07:00-09:00)

      • Banner Upgrades:
        - DB Side (08:00 – 11:00)
        - BA FINAID 8.43.2.2 & BA WWW-FINAID 8.43.2.2
        - Banner General 8.15
        - Banner Admin Common 9.3.30
        - Banner Student 8.26 (Dependency for Banner Financial Aid 8.44)
        - Banner Financial Aid 8.44
        - Banner Financial Aid 9.3.33
        - Banner Financial Aid 8.44.1
        - Banner Financial Aid 9.3.34
        - Banner Ethos API DB Upgrade 9.29 (Mark)
        - Student API 9.29
        - Application Side (11:00 – 15:00)
        - Banner Admin Common 9.3.30
        - Student API 9.29

    • We will make two admin servers and two SSB servers available as soon as possible, and I will send an email as a reply to this thread once this is completed. But please plan on an outage for the whole period.
      The outage is affecting all Banner Apps. App Nav/ Admin Pages/ BCM / BDM / NelNet/ JobSub/ Appman/ Student SSB/ Faculty SSB / General SSB / Banner Extensibility (Page Builder)/ uAchieve CSU and CSUP/ Registration SSB

  • May 4, 2023 BANTEST - Banner Upgrades applied on BANTEST tomorrow 5 HRs outage

    • To keep BANTEST matching BANPROD (product version-wise), and instead of cloning it during next week during Finals week, we plan to apply all the updates that are going to happen Sunday in Production to BANTEST tomorrow starting 09:00. We expect the upgrades will take 5 HRs. 

      Please expect an outage to the following products in BANTEST starting at 09:00 and ending around 14:00: App Nav/ Admin Pages/ Job Sub/ BCM / BDM/ Nelnet/ Student SSB/ Registration SSB / Faculty SSB/ General SSB / Banner Extensibility / uAchieve. 

  • May 1, 2023 EEID vulnerability - Production deployment today 17:00

    • Mitigation for a new vulnerability that is impacting our EEID clusters need to be implemented asap. Scot deployed the mitigation in our test cluster, and he is testing it now. We would like to implement this mitigation into Production this evening starting 17:00.

    • It will require a rolling restart of our EEID cluster. Impact should be minimal but, campus may find that their attempt to authenticate will fail and will need to retry.

  • Apr 28, 2023 Banner 8 server / ORDS restart today 17:00 (15 min outage)

    • We must restart the Banner 8 server (ORDS) to finish the application server work for CourseLeaf CLSS implementation. The impact is minimal. Only Banner 8, which includes ORDS, will experience an outage for 15 min starting at 17:00.

  • Apr 21, 2023 Network maintenance on core routers. - Date/Time: Friday, April 21, 2023, 8 pm – Saturday, April 22, 2023, 4 am

    • Details:  Maintenance will be performed on our primary router, which provides connectivity to the internet.  This work will be performed in two phases:

      • Phase 1 - All workloads will be flipped to the redundant side on the border routers, Palo Alto firewalls, and the data center firewalls so traffic and services will be available as usual.  First, the code on the router will be upgraded, a system control board will be replaced, and we will install new 100GB interface cards.  This is expected to take 4 ½ - 5 hours when no impact is expected as services will continue to run on the redundant side.

      • Phase 2 - After work is complete on the primary side, workloads will be flipped back to the primary side to test the new 100GB connections.  Once we fail back to the primary side, there could be potential outages if the new connections don’t come up correctly.  Assuming maintenance is successful at this point, the team plans to initiate multiple failovers to test our improved redundancy design – and at that point, further maintenance should be largely transparent.

    • This is a foundational upgrade that is both necessary and will enable future work to refresh the core of our network.  Thank you to everyone for your support, and as usual, please reach out to noc@colostate.edu if you encounter an outage in your environment beyond our maintenance window.

    • Impact:  During the second phase of the maintenance, the potential for temporary outages in internet connectivity, including connectivity to CSU Pueblo, exists.  We expect this work to be concluded by 4 am and plan to have the network in a steady state for the weekend’s business as soon as possible.

  • Apr 16, 2023 BANPROD Sunday Maintenance 7:30 AM - 11:00 AM

    • To resolve some issues introduced after Q3 upgrades, we need to deploy a new version of Student SSB (Student Self Service 9.21.0.7). There are a couple of dependencies for this new version of Student SSB. Banner Student 8.25.2 and Banner 9x Database Upgrade 9.29.
      We will use the Sunday maintenance period to deploy the above products. Since there is a DB component, Banner DB needs to be shut down.

    • All Banner applications will be down during the deployment time. We expect to start at 7:30 AM; Banner Admin Pages and SSB will be available for regular usage starting 11:00 AM Sunday.

  • Apr 15, 2023 Exadata Maintenance on April 15 at 9:00 am MDT → Completed

    • There is scheduled maintenance this Saturday, April 15 beginning at 9 am Mountain Daylight Time. This is an infrastructure update so no new versions of the database or underlying OS versions will be applied.  The time required for the maintenance window is estimated to be approximately 7 hours.  This maintenance is performed by Oracle and we have no control over the process.  

    • The databases servers will be updated in a rolling fashion.  This means that the databases will remain available, and all Banner Applications except BCM supports rolling restarts.

    • In short, we don’t expect any downtime during this maintenance period on Banner Applications. We are going to have only one server of BCM running, and both BCM servers will be restarted and be running Sunday after Sunday maintenance period (look for my second email regarding Sunday once we got clearance from users)

    • Appman doesn’t support rolling upgrades so Appman will be down during this maintenance period.

  • Apr 10, 2023 BANTEST Clone → Completed

    • BANTEST won’t be available today during the clone. Expected to be back in business 04/11

  • Mar 26, 2023 AMPROD NFS migration to Powerstore - Banner Apps - Admin Pages / Nelnetl / All Banner SSB / BCM will be out of service between 7:00 AM to 10:00 AM - Completed

  • Mar 18, 2023 2022 Q3 Banner Upgrades - Completed

    • Impact all Banner applications (Admin Pages/ App Nav/ BCM / Nelnet / BDM/ BAM/ Jasper/ Student SSB / General SSB / Registration SSB / Banner Extensibility/ Faculty SSB/ uAchieve)

  • Mar 12, 2023 BANPROD Admin / SSB app servers Restart Sunday 8:00 -9:30 - Completed

    • We plan to use the maintenance period this Sunday, 03/12/2023, to restart Admin and SSB app servers. We want to ensure all changes applied to the firewall and db drivers are in total effect for all apps. We expect downtime between 08:00 – 09:30.
      Both admin servers, including Nelnet and BCM, plus two SSB Sevres, will be up and running by 09:30, and we will add the rest of the SSB servers when they are ready right after.
      Please let us know if you have any questions or concerns.

  • Mar 7, 2023 BANPROD Admin / SSB Maintenance 03/07 [6:00 - 7:00]

    • Admin Servers and SSB servers’ certificates are going to expire on 03/11. We need to renew them before that date.

    • We plan to renew the certificates on all servers on 03/07 – 06:00 AM. Admin / SSB / Nelnet will be down starting 6:00 AM. Admin Servers / Nelnet / and two SSB servers will be available for all at 07:00 AM, and we will continue to add the rest of the SSB servers immediately.  

    • In short, Admin Pages / Banner SSBs / Nelnet will be out of service 07/03 for one hour starting 6:00 AM.

  • Feb 28, 2023 [Current] BANPROD DB Restart 14:00 - (14:30 - 15:00)

    • We figure out the issue causing timeouts for admin servers and other services. To fix that, we need to take the DB VMs down. We plan to do that today at 14:00. And we expect downtime will be between 30 to 60 min.  

    • This process is going to affect Admin Pages, all SSB apps, uAchieve, BCM, Appman, Jobsub, Workflow, Nelnet, Jasper Transfer Bridge, and any other Banner-connected Apps. 

  • Feb 27, 2023 BANPROD DB Restart 14:00 - 14:30 - Completed

    • Due to multiple connectivity issues reported today, we need to restart Banner DB to fix the problems permanently. We plan to start the restart process at 14:00, which is estimated to be completed by 14:30.

    • This will affect Admin Pages/ SSB apps (Student / Faculty / General / Extensibility) / uAchieve Fort Collins / BCM / All Banner connected app.

  • Feb 26, 2023 Exadata OS and DB upgrade - Downtime 06:00 - 12:30 - Completed

    • Impact all Banner applications (Admin Pages/ App Nav/ BCM / Nelnet / BDM/ BAM/ Jasper/ Student SSB / General SSB / Registration SSB / Banner Extensibility/ Faculty SSB/ uAchieve)

  • Feb 12, 2023 BANPROD - OS / Banner Security Patches / Workflow Upgrade / The transcript importer deployment for uAchieve - 02/12 - 0700 - 1330

    • Impact all Banner applications (Admin Pages/ App Nav/ BCM / Nelnet / BDM/ BAM/ Jasper/ Student SSB / General SSB / Registration SSB / Banner Extensibility/ Faculty SSB/ uAchieve)

  • Feb 1, 2023 BANTEST is back. Exadata Upgrades + Banenr Q3 upgrades.

    • A new clone was performed to BANTEST, which is on the upgraded version of Exadata, and Banner Q3 upgrades with the needed security patches have also been applied. OS patches are also applied to BANTEST.

  • Jan 11, 2023 BANPROD Admin Servers Maintenance 1700 - 1830

    • We plan to work on a couple of items today between 1700 and 1830:

      • Jasper upgrade – we figured out a resolution to the issue of the Jasper Appman Bridge

      • Fix BAM access issue resulting from yesterday's work
        Impact:
        The work is on Admin Servers only, so there is no impact on SSB applications. The following products are going to be down during this time window:
        Admin Pages/ App Nav / Nelnet/ BCM/ BAM

  • Jan 10, 2023 BANPROD maintenance Tuesday 1700 to 1830

    • To apply a fix regarding GOAFGAC, Banner DB (BANPROD) needs to be bounced, which means we need to restart all Banner applications. → Completed

    • Jasper Server Upgrade → Issues and rolled back

      We plan to implement the fix Today after working hours 1700, and we expect we need 90 min. So all applications below won’t be available between 1700 and 1830.

      Admin Pages/ App Nav / BCM / BAM / Jobsub / Workflow/ Nelnet/ Student SSB / Registration SSB/ Faculty SSB / Banner Extensibility / General SSB / uAchieve / Jasper

  • Jan 9, 2023 BANTEST Admin Servers Maintenance 15:00 - 16:00 → Completed

    • We will upgrade Jasper on BANTEST today starting 1500, and we expect we will be done by 1600. The following apps won’t be available on BANTEST during this time: 

      Admin Pages / App Nav / BAM / BCM / Nelnet/ Jasper

  • Jan 9, 2023 BANUPGD Q3 Upgrades. - Completed and ready for stakeholders testing

  • Dec 28, 2022 BANTEST maintenance today 1:00 PM - 2:00 PM

  • Dec 14, 2022 - Dec 15, 2022 17:00 - BANPROD - Implement Critical Grails security patch on BANPROD - All Admin / SSB apps are affected. This will be done in rolling fashion (1 Admin server / 5 SSB servers at a time) - Completed

  • Dec 16, 2022 Widespread issue with Shibboleth authentication this morning - Resolution in progress - Please check this link for real time updates.

  • Dec 12, 2022 BANTEST - Critical Grails security patch - All Admin / SSB apps are affected. - Completed

  • Dec 10, 2022 BANPROD - FINAID upgrade / FGE fix - Completed

  • Dec 5, 2022 FINAID Upgrade on BANTEST - Completed

  • Dec 1, 2022 PROD BDM Server Restart 17:00 - Completed

  • Dec 1, 2022 BANDEVL Clone - Application of MFA (Duoe Factor Authinitication)