Loading...
ITEM 2 IT issues and strategiesITEM 2 CITY OF se o MINNESOTA DATE: 4 February 2015 TO: Mayor and City Council Members Lori Johnson, City Administrator MEMORANDUM FROM: Daniel Jordet, Administrative Services Director RE: Information Technology (IT) Issues and Strategies In order to prepare for a discussion of the current condition of the IT function of the City I am preparing this memorandum. It includes a number of points raised in conversations between October 2014 and February 2015. These conversations include meetings between our IT consultant Rob Snavely, the City Administrator and myself. In addition, some of the issues were discussed at the January 14, 2015 Administrative Subcommittee meeting. The memorandum will attempt to summarize the discussions, recommendations or conclusions and outstanding issues. Security and Continuity of e-mail Service: In the spring of 2014 the e-mail system of the City went out for an extended period. It happened, unfortunately, when our consultant was out-of-state on a well- earned vacation. This raised two issues; the first being the operation and maintenance of the e-mail system in an efficient and effective way and the second being the issue of who will provide backup services when our primary consultant is not available. The second issue will be addressed later in this memo. It was discovered that a computer not belonging to the City but having access to its network through the Deputy Sherriff's office in the Prairie Center was the culprit. It took about a week and a half to find and fix the problem so that City e-mails could again flow freely and be received properly. Anti-virus monitoring for incoming e-mails is already in place for the City. Securance, through US Internet, scans incoming e-mails for virus infections and deals with problems. Outbound e-mails are not treated in the same manner. For an additional charge the outgoing e-mails could be scanned as well with the entire inbound/outbound virus scanning process costing less than $ 100.00 per month. Another method for dealing with security and possible downtime on our e-mail server is to contract the entire service to Securance for hosting, backup and virus protection of our e-mails. While this service would cost the City something more in the range of $ 225 to $ 275 per month, depending on the number of users, it would eliminate the need for having our own server and anti-virus protections. It would be recommended that staff obtain costing on the alternate ways of providing secure e-mail service and present a concrete proposal to the Council including comparative costs and advantages or disadvantages of any one service over another. Archiving of e-mails is another issue to be addressed. Minnesota law indicates that e-mails must be archived for public inquiries. That is currently being done on-site at City Hall in a format that is legally correct but very difficult to search for information recovery. This would likely cost an additional $ 100 or so per month. General Continuity of Service: In order to assure continuity of municipal services with minimal downtime, a question was raised about how long the City could tolerate a shutdown of its entire computer system. Would we need to be back up and operating within one day? Would we be able to tolerate 4 hours of downtime before systems were restored? Should there be redundant systems so that in the case of a system failure a backup system would automatically kick in? Discussions indicated that a full day was too long to be completely down, including e-mail, licensing programs, financial software, building permit software, access to working files for Word, Excel, PowerPoint and Adobe and any other network hosted software used by staff. The cost/benefit of having redundant systems available for full continuity of service would likely be too high to justify the benefit. It was suggested that a plan for recovery of all systems within a time frame of up to 4 hours would be an appropriate time frame for which to plan. In order to assure this level of service availability there are hardware issues to be addressed and consultant service/service backup issues to be addressed. On the hardware side, the City has operated its current servers have been operating continuously for five years but still seem to be functioning well. In the short term they do not present failure risk. However, within the next year or two we will have to consider server replacements. New servers will likely have larger and faster capacities. Rob has also introduced the idea of structuring the new server systems as "virtual servers" within more standardized equipment. In addition, the modem and router currently in use have much higher risks of failure. Replacing the router and obtaining and configuring a backup router are being addressed. Comcast should be contacted for information about replacing the modem with something more up-to- date and perhaps keeping the old one as a backup. Charter should also be approached about running the cable to the City Hall location so that modem failure would not stop work in the building. Backups of the entire system, relevant programs and work files are done on a regular basis by our consultant. In addition to this data backup, it is recommended that during the periodic cleaning and servicing of each individual work station, a mirror image of the hard drive in that computer be made, stored and updated at least quarterly in order to reduce downtime should a malfunction occur. A "spare" computer pre -loaded with all operations programs required within a work station system should be obtained to function as a replacement or fill in workstation computer in the case of an egregious malfunction of any work station. Backup to service by our consultant is another issue that has been discussed. TR Computer of White Bear Lake has been available to the City as a backup provider of services for some time. However, communication and familiarity between City staff and TR staff has been minimal. TR has been involved with City systems in replacement, upgrade and troubleshooting in the past. These contacts have always been through our consultant. It is suggested that forming a clearer and more informed relationship with TR Computers will provide the backup service safety net that is required for the City. I.T. Planning Rob Snavely 2015-01-30 Current operations • Servers: Our existing servers are 5 years old and are made by SuperMicro. They are XEON based, server -class hardware with hot-swappable RAID drives. At the time of their purchase, the hard drive and RAM sizes were considerable — but not by today's standards. Both servers are identical. Hard drive capacity is at about 50%. o Drive(s) C = 150GB, o Drive(s) D = 300GB, o RAM = 12GB • Server OS: Windows Small Business Server 2008 with Exchange 2007 • Email: We are doing on-site MS Exchange hosting using Microsoft Exchange 2007. The advantage of this is unlimited mailboxes at no additional cost. Our incoming email is filtered through Securence (US Internet). There is no outgoing email filtering. • Network: Our network uses 100MB switches (both powered and standard) with backup switches on-site. Our Internet business service is provided through Charter, and includes five static IP addresses. The main line comes into the Prairie Center where we have the modem and router. The signal is transmitted to the main building via radio. When the primary function of the (now) city hall was public works, little network traffic crossed the radio, but now it's the main link to the outside world for most city employees. The modem is owned by .Charter. We do not have a backup modem, but Charter will bring a replacement within hours should it fail. During that time we would not have Internet or outside email services. Our router is very outdated and has not had D-Link support for a number of years. We don't have a replacement router, so if it should fail, we would be without most computer services until it could be replaced and configured. • Backups: Every night, each computer backs up its User folders (as well as any other special folders specific to that computer) in a password -protected ZIP file and sends it to a USB HDD on the primary server. Each week, those files are moved to a folder with the current date, effectively making a snapshot of that week's data. That USB HDD is then swapped with an identical one and taken off --site. This allows us to retrieve files many years old. Every night, the servers back up their modified data using Symantec Backup Exec. Each week, an off-site USB HDD is updated to match the on-site drives and then returned off-site. Each month the servers make a full backup of their entire data. • Antivirus: Currently, all computers are using the AVG anti-virus program. Randomly, computers are also checked with MalwareBytes anti-malware and SpyBot's Search and Destroy programs. • Support: Although I'm able to handle the vast majority of day-to-day IT services, there are times when outside help is needed. To date, I've been using the services of TR Computers based in White Bear Lake. We purchased the server hardware through them and are also using their PermitWorks software. They are already intimately familiar with the city's hardware and software and have always been available immediately when needed. Short-term, or `easy' fixes • Servers: Our current servers are running well, and I don't see any short-term issues • Email: A relatively simple fix to the SPAM blacklisting that occurred last spring would be to implement outgoing email filtering via Securence at a cost of 50c per month per email account. Federal requirements state that we need to archive email for a number of years. I am currently doing this through capturing cached email on each computer and storing that on a backup drive. Although it works, it would be very cumbersome to retrieve any specific email. Securence also offers 10 -year incoming/outgoing email archiving for $4.00 per user per month. This would be far superior to the method currently being used. • Network: Our router should be replaced with a current one. Ideally we would purchase two identical ones and configure them the same. This would give us minimal down-time in case of a router failure • Backups: For a one-time cost of approximately $35 per computer plus a USB HDD, I would recommend that we create a mirror of each actively used computer once a quarter (or so). This would allow bare -metal recovery of any mirrored system in much less time than it takes now to prepare a new system after a failure. I recommend that we setup one computer with all the typically -used software on it (PermitWorks, Clarity, Microsoft Office, etc) that in the event of someone's computer failing, this one could be a temporary drop-in replacement allowing them to continue to work while the problem is being corrected. Strategic planning • Servers: The city's computer needs have grown tremendously in the past few years. Building permits, finances, utilities, records, email all require the use of a working server. So when we replace our servers in the future, it seems that the main question to be asked is, if our server goes down, how long can the city be without a server? One day? Four hours? One hour? That question needs to be answered to determining the future hardware/software. One day: (The cheapest option) Similar, but updated, hardware to that currently being used. If our current server fails, it can be replaced and the backup data restored. It takes time to do this, but we've already demonstrated that our backup system works and allows us to restore operations within a day. Approximate cost $8000 Four hours: Using a high-end, dual XEON Dell server with 64 GB RAM and a RAID -10 HDD controller we could virtualize the domain controller server and the application server. This would allow for easy replication of the server onto other platforms if the server hardware went down. The hardware would cost approximately $8500 and the setup and configuration approximately $4500. One hour: This would use two high-end Dell servers and a SAN (Storage Area Network). The SAN allows both servers to share backup storage such that if one server fails, the other can be started from the commonly shared SAN. This is an expensive option: between $25,000 to $50,000 for the hardware alone, but would ensure any downtime would be extremely short. • Server OS: Windows Server 2012, Windows SBS has been replaced by Windows Server Essentials and no longer includes Exchange. • Email: If we choose to have our email hosted by an off-site service, we gain a number of advantages: incoming and outgoing filtering and email redundancy. US Internet offers the service for $8.95 per user per month and TR Computer offers a similar service for approximately $4.00 per user per month. • Network: It might be worth getting a price quote to bring Charter cable to the new city hall building. We should replace our current router with a newer one — preferably two. • Backups: There are numerous backup possibilities. Although cloud -based backups are popular, one thing to consider is that in the loss of the Internet, backup data is not available. One method of backup would be the DATTO system which would not only allow local and off-site backups, but the system can also double as a reduced -resource virtual server. This would allow operations to continue (at a reduced speed) while a replacement server is being prepared. A variety of models are available with differing capabilities. The s1000 model costs approximately $2500 Hardware / Software Dell PowerEdge T420 1 $6,500.00 $6,500.00 Dual Xeon 2.5, 64 GB RAM, Raid 10 controller, (6) 1 $995.00 $995.00 600 GB SAS - Raid 10, DVD, Dual Intel NIC, (2) hot 12.5 $125.00 $1,562.50 swap power supplies, Security Bezels, 5 year Dell 2 $110.00 $220.00 Pro Support Windows 2012 Server- Hyper V, DC (W2012), 1 $883.00 $883.00 Application Server (W2012) 5 $125.00 $625.00 Windows 2012 Server Client Access Licenses 25 $34.00 $850.00 Dell Shipping 1 $24.00 $24.00 HW/SW TOTAL NOT INCLUDING OPTIONAL PURCHASES AND TAX: $8,257.00 Estimated Labor: Network File Server Installation - Hyper V Role / 10 $125.00 $1,250.00 Windows 2012 R2 DC 1 $995.00 $995.00 Join workstations to the domain and copy profiles - 12.5 $125.00 $1,562.50 30 minutes per PC 2 $110.00 $220.00 *** based on 25 computers *** W2012server- application server / laserfiche 4 $125.00 $500.00 Transfer data from the old server (ESTIMATE) 5 $125.00 $625.00 Configure Network Printer (1/2 hour per printer) 1 $125.00 $125.00 Configure Network, File system & training 1 $125.00 $125.00 Reconfigure Backups - DC / W2008Server 3 $125.00 $375.00 Reconfigure Multifunction Copier/ScannerSMB 1 $125.00 $125.00 Addressbook and Relay Condigure UPS / Powerchute battery backup settings 1 $125.00 $125.00 LABOR TOTAL NOT INCLUDING OPTIONAL PURCHASES AND TAX: $4,812.50 ENGAGEMENT TOTAL NOT INCLUDING OPTIONAL PURCHASES AND TAX: $13,069.50 OPTIONAL PURCHASES: Hardware / Software LOOK@IT Server monitoring (3 servers) 1 $160.00 $160.00 Shadow Protect Virtual Server (3 Pack) w/ 1 yr. 1 $995.00 $995.00 support Seagate Backup Plus Slim 1 TB - Onsite / Offsite 2 $110.00 $220.00 Hard drives Server Cabinet with Shelves 1 $1,500.00 $1,500.00 Upgrade to Dell Rack mount server 1 $800.00 $800.00 SmartUPS 1500VA (w/ Powerchute) 1 $549.00 $549.00 Estimated Labor: Configure existing software - Permitworks, 6 $125.00 $750.00 Laserfiche. *** Discuss any other software *** Configure Backup 3 $125.00 $375.00 The prices quoted above are at today's market value. Prices and availability will be verified at time of purchas MN State tax laws require us to charge you tax an all purchases as well as service to install any new hardware or software. This tax will be assessed on all purchases and only the appropriate services provided. Hardware / Software Microsoft Hosted Exchange - 0365 exchange only 30 $4.00 $120.00 (billed monthly) HW/SW TOTAL NOT INCLUDING OPTIONAL PURCHASES AND TAX: $120.00 Estimated Labor: Labor to Configure 0365 hosted exchange 4 $125.00 $500.00 Transfer e-mail / migrate to the cloud 5 $125.00 $625.00 (Estimate - will want to clean up any mailboxes) Reconfigure Outlook Profiles -30 minutes per 15 $125.00 $1,875.00 mailbox (Estimate) Reconfigure Phones, (pads, tablets ect...-15 4 $125.00 $500.00 minutes per device (Estimate) LABOR TOTAL NOT INCLUDING OPTIONAL PURCHASES AND TAX: $3,500.00 ENGAGEMENT TOTAL NOT INCLUDING OPTIONAL PURCHASES AND TAX: $3,620.00 The prices quoted above are at today's market value. Prices and availability will be verified at time of purchas MN State tax laws require us to charge you tax an all purchases as well as service to install any new hardware or software. This tax will be assessed on all purchases and only the appropriate services provided. Alf taxes will be added and included on the invoice. Our estimate is based on our prior experience multiplied by our standard hourly charges for these services, as summarized on the System Specification section. Any unusual disruption in the implementation process may result in higher fees. On-going support will be provided on an "as needed" basis, at your request. Consider purchasing telephone technical support contracts with the software vendors that offer them, if you require immediate response or guaranteed call back times. The terms for any additional hardware or software not specifically listed in this engagement are due upon receipt of invoice, if the order total is less than $1,000. Pre -payment is required for orders exceeding $1,000. We invoice our labor monthly, separate from computer hardware and software, and expect payment within thirty (30) days. We expressly reserve the right to assess a finance charge at the rate of 1.5% per month on any balance on your account that is older than thirty (30) days. You agree that in the event it becomes necessary for this account to be referred to an attorney for collection (whether or not suit is commenced), you will be responsible for payment of all reasonable costs of such collections, including reasonable attorneys' fees. Each monthly invoice will include a detailed description of the services provided. Our fees are based on the hourly rates of the various staff level and are adjusted annually for general economic factors. We schedule staff to serve your company based on the level of tasks to assure that you receive cost-effective service. TRCS HOURLY RATES Our hourly rates are $125 for labor performed Monday through Friday from 8 a.m. to 5 p.m. Labor performed after 5 p.m. or on the weekends will be billed at $182.50 per hour (time and a half). WARRANTIES AND LIMITATIONS