Extended Character Sets for UCAS Systems

Transcription

1 Extended Character Sets for UCAS Systems Admissions Conference 2010 Mike Gwyer

2

3 ASCII The American Standard Code for Information Interchange A character-encoding scheme based on the ordering of the English alphabet. US-ASCII is the IANA preferred charset name for ASCII. ASCII includes definitions for 128 characters 33 are non-printing control characters 94 are printable characters space is considered an invisible graphic. The most commonly used character encoding on the World Wide Web was US-ASCII until December 2007, when it was surpassed by UTF-8

4 Unicode and the UCS Unicode and the ISO/IEC Universal Character Set (UCS) have a much wider array of characters Unicode and the UCS support more characters by separating the concepts of unique identification Using natural numbers called code points Encoding (to 8-, 16- or 32-bit binary formats, called UTF-8, UTF-16 and UTF-32). The 128 ASCII and 256 ISO (Latin 1) characters are assigned Unicode/UCS code points that are the same as their codes in the earlier standards. ASCII can be considered a 7-bit encoding scheme for a very small subset of Unicode/UCS,

5 Oracle character sets Single-Byte Encoding Schemes Efficient. Take up the least amount of space to represent characters Easy to process and program - one character can be represented in one byte. Classified as one of the following: 7-bit encoding schemes - up to 128 characters and normally support just one language. 8-bit encoding schemes - up to 256 characters and often support a group of related languages. Multi-byte Encoding Schemes Needed to support ideographic scripts used in Asian languages like Chinese or Japanese because these languages use thousands of characters. Use either a fixed or a variable number of bytes to represent each character. Fixed-width multi-byte encoding schemes - each character is represented by a fixed number of bytes. Variable-width multi-byte encoding schemes - uses one or more bytes to represent a single character.

6 Oracle character sets (continued) Length Semantics In single-byte character sets, number of bytes = number of characters In multibyte character sets, a character or code point consists of one or more bytes. Calculating the number of characters based on byte lengths can be difficult in a variable-width character set. Calculating column lengths in bytes is called byte semantics, while measuring column lengths in characters is called character semantics. Character semantics useful for defining the storage requirements for multibyte strings of varying widths. For example in a Unicode database (AL32UTF8), suppose that you need to define a VARCHAR2 column that can store up to five Chinese characters together with five English characters. Using byte semantics, this column requires 15 bytes for the Chinese characters, which are three bytes long, and 5 bytes for the English characters, which are one byte long, for a total of 20 bytes. Using character semantics, the column requires 10 characters.

7 At UCAS.. AL32UTF8 Oracle Database character set that is appropriate for XMLType data. Equivalent to the IANA registered standard UTF-8 encoding Supports all valid XML characters. 1-4 bytes per character Continue supporting future Unicode standard Not recognized by pre-9i database Apart from UKPASS, UCAS Systems currently have database columns created with Characters, not Bytes... Which means a 30 char field might only be able to hold 7 characters...

8 UCAS wishes to provide applicants with the ability to enter selected details using the full range of characters available to and other languages

9 This is seen as necessary to fulfil our responsibilities to comply with the Welsh Language Act UCAS has issued a joint statement with HESA, setting out our intention to adopt the emerging MIAP standards In liaison with UCISA-CISG, UCAS has now twice canvassed the HEI community to determine their readiness to accept applications containing extended characters

10

11 The UCAS web site, on-screen instructions within the Apply application and all help text is already available in Welsh Any applicant who is applying exclusively to Welsh universities may choose to complete their application in Welsh text (but currently only using the standard characters of the English alphabet) Applicants can ask for all correspondence in relation to their application to be written in Welsh

12 The Welsh language allows for the use of 28 accented characters, in addition to the Standard English character set The following accents can appear on both upper and lower case characters A ; E ; I ; O ; U ; W and Y Circumflex ^ Acute Grave Diaeresis `

13

14 During 2008, the hope was that the extended character set could be implemented for the YOE 2010 cycle, starting summer On 05 November 2008, a questionnaire was sent via the UCAS technical mailbase to all participating HEIs to establish their readiness.

15 Q1a ~ What is the current state of readiness within your institution for the implementation of Extended Character Sets for the 2010 entry cycle? Will NOT be ready 36.4 % Will be ready 15.1 % Don t know 48.5 %

16 Q1b ~ Please indicate if the level of readiness varies depending on the scheme ~ UCAS ; GTTR ; CUKAS ; UKPASS? Respondents overwhelmingly stated that their level of readiness did NOT vary depending on scheme Four did not answer ; All others said that there was no dependence on scheme

17 Q2 ~ Is your institution actively working towards a UNICODE implementation for extended character sets? Don t know 7.1% No Will NOT be ready 28.6% 36.4 % Yes 64.3%

18 Q3 ~ When do you believe your institution will be ready to accommodate a UNICODE implementation for extended character sets? Never 3.4% Beyond % Don t know 7.1% % 17.9% %

19 Q4 ~ If you answered Never to Q3, how do you plan to accommodate Extended Character Sets within your institution? a) Currently no plans to implement extended character sets b) Uncertain c) None Unicode implementation e.g. ISO (also known as Latin-8 or Celtic All respondents who answered Never to Q3, answered a) No plans to this question

20 Q5 ~ Please indicate what assistance UCAS could provide your institution to aid the smooth implementation of Extended Character Sets? Provide Help Create Pilot Year software a financial a test parallel Character Forum specification data provider data support Map Capture Provide Establish Avoid a Big and a forum explanation Bang store for both implementation. institutions of Standard the mapping and Run stakeholders Unicode between a parallel to characters exchange Unicode characters version views, allowing alongside comment and each the institution Live and equivalent question for a to whole choose Standard each cycle other, characters which UCAS, version UCISA on they extended received character set issues

21

22 In January 2009 the UCAS Change User Group decided : Which Schemes and When The proposal is that ALL of the current online application systems will be modified to enable them to accept Extended Characters for the 2012 Entry cycle (i.e. starting Summer 2011) (UCAS / GTTR / CUKAS / UKPASS)

23 Which Character set? Following the implementation and in line with the MIAP Common Data Definitions guidelines, online applicants will be able to enter characters from any of the following character sets : Basic Latin (excluding C0 control characters) Latin-1 (excluding C1 control characters) Latin Extended A Latin Extended B Latin Extended Additional

24 Which Data Items? Should extended characters be acceptable in all free-text data items? Or, should we consider initially restricting input to specific fields? E.g. Surname ; Forename? Addresses? Personal statement? UCAS is currently working with UCISA-CISG to finalise this decision

25 Test Data The first application to go live in each cycle is CUKAS, which is typically launched in June. In order to be ready for a launch in June 2011, we should aim to make ODBC-Link and XML-Link test data available to the HEIs and Software houses by January 2011 (or sooner).

26

27 Implementation of extended characters is likely to impact on many aspects of data processing Printing Sorting Field lengths (One extended char > 1 byte) Matching e.g. applicant de-duplication Smith vs Smîth Accessibility e.g. Text to speech software Complexity of input using standard keyboard

28

29 Gather feedback from Software houses Finalise decision on which fields Create dedicated extended character set test environments ~ ODBC-Link and XML-Link.

30 Disquiet regarding decision to enable all fields Concern over downstream implications HEI s cost/resource implications In house developer with 75 separate systems. Estimated upwards of 100,000 for implementation Phased implementation more acceptable due to downstream data transfer implications. UKBA implications Cannot cater for Unicode in their student records system. have no plans to accept Extended character sets usability issues Searching, filtering and sort lists Ability to enter these through a UK keyboard

31 Some ready... Will be Unicode compliant by September 2010 Others not Planned resource cannot begin working on this scope has not been defined Will now overrun their projected timescale will not be ready to implement

32 MIAP cannot implement the standard there is currently no change in SLC not currently expected to be ready to accept Unicode Awarding Bodies Legal implications changing the meaning of words input in substituting an extended character for an ASCI character. UCAS is currently liaising with solicitors to establish compliance around the issue Current advice is that if institutions are unable to accept Unicode characters these must be translated into a? rather than try and find the next reasonable alternative. Simple from a UCAS perspective but

33 Not implement Unicode due to market demand / issues with other bodies (SLC/MIAP/Awarding bodies) Implement a solution for 2012 where a select number of fields are collected in both ASCII and Unicode Full implementation for 2012 as proposed by the CUG.

34 A small number of fields would be collected using Apply in both ASCII and Unicode formats. Both versions of these fields would be stored at UCAS and made available to HEI through Web-Link, XML-Link and ODBC-Link. This would allow Institution to which version of the field to collect based on their Unicode readiness.

35 Which fields would permit ASCII & Unicode versions Applicant name Applicant Address (not postcode) Personal Statement Reference

36 Applicant name Applicant Address (not postcode) All applicants would be permitted to enter their name and address with extended characters but would be prompted for an ASCII equivalent where necessary. Personal Statement Reference Applicants that select the Welsh Language option within our Apply product would be permitted to write both an English, ASCII, and Welsh, Unicode, version of their personal statement.

37 Web-Link would show both ASCII and Unicode versions of all fields this could be changed based on Institution preference XML-Link would have new Unicode GET methods for relevant applicant details that would return the Unicode version of the data items ODBC-Link would have views amended, based on Institution preference, to show either the ASCII or Unicode version of the data items

38 Applicant is able to decide how their ASCII version of their data is entered no automatic conversion without them knowing / legal challenges Welsh applicants would now be able to apply with full Welsh personal statements and references to both Welsh and non-welsh Institutions with a Welsh personal statement and reference. Institutions would be able to decide when they are ready to handle Unicode character sets UCAS could monitor the adoption of Unicode enable UCAS to make an informed decision on : When to increase the number of fields collected using Unicode When to removed the need for the ASCII equivalent

39 It does not cover all the fields as suggested by the CUG It does require increased storage at UCAS although not many applicants choose the Welsh options within Apply It does add some complexity to UCAS Link products This option would not scale to many fields we could not do this for the whole application user interface and storage issues would make this impossible

40