Appirio's Tech Blog

Friday, February 5, 2010

Displaying Currencies In a SFDC Formula Field or Visualforce

Glenn Weinstein

I wanted to put a Formula field on the PSE Project detail page to show a red or green flag next to a currency amount, based on some criteria.

The flag image is the easy part.  The hard part is displaying the currency amount with proper formatting (commas and decimal points).  The TEXT() formula function results in a straight number, e.g. $1,532,311.78 turns into 1532311.78.

After much research, I determined there's no easy way to format numbers in conjunction with TEXT().  So I resorted to brute force, and wrote my own formula.  After much trial-and-error, I'm pretty sure I've perfected it. 

IF(
  someCurrencyField__c >= 1000000,
  TEXT(FLOOR(someCurrencyField__c / 1000000)) & ",",
  "") &
IF(
  someCurrencyField__c >= 1000,
  RIGHT(TEXT(FLOOR(someCurrencyField__c / 1000)), 3) & ",",
  "") &
RIGHT(TEXT(FLOOR(someCurrencyField__c)), 3) & "." &
IF(
  MOD(someCurrencyField__c , 1) * 100 < 10,
  "0" & TEXT(ROUND(MOD(someCurrencyField__c , 1), 2) * 100),
  TEXT(MIN(ROUND(MOD(someCurrencyField__c , 1), 2) * 100, 99))
)

Just replace someCurrencyField with your currency field, and you'll get back a nicely formatted number with 2 decimal points.  Notes:

  • Works for numbers up to 999,999,999.99.  (If you need higher, just add another IF() clause.)
  • Formula above assumes a multi-currency org.  If not, replace CurrencyIsoCode with a hard "$" string.
  • The last IF() clause is particularly tricky, to ensure we always get 2 digits to the right of the decimal point. 
  • That final MIN() is to guard against situations where someCurrencyField extends to more than 2 decimal points, to avoid inadvertently rounding up to 100 (we don't want, for example, $56,314.9999999 to come out as $56,314.100).
If you're curious, I use the above snippet in a formula that includes a red or green flag, based on some criteria.  Here is the (anonymized) formula, along with the IMAGE() formula to display the flag:

IF(
  something__c > somethingElse__c,
  IMAGE("/img/samples/flag_red.gif", "Red"),
  IMAGE("/img/samples/flag_green.gif", "Green")
  ) & " "  &
TEXT(CurrencyIsoCode) & " " &
/* copy in currency display code from above */

Then, pull the source field off your page layout, and drop in your new formula field.  This technique adds some visual enhancement to otherwise-tedious looking displays of numbers:

Capture

I like putting the flags inline with the numbers - the flags help you quickly assess whether a number is "good" or "bad".  And they look great as a column in a View list.

Note that you can run into a similar issue with Visualforce merge fields.  And if you're using Visualforce for an email template, you can't even rely on solving formatting issues via the controller.  But there is a much more straightforward solution to currency formatting in Visualforce, using the apex:outputText tag in conjunction with apex:param.  Here's an example:

<apex:outputText value="{0, number, ###,###,###,##0}">
  <apex:param value="{!relatedTo.someCurrencyField__c}"/>
</apex:outputText>

Per the Visualforce Developer's Guide, you can use the same syntax as the Java MessageFormat class for your apex:outputText formatting string.  Also, if you're showing currency (as I am here), you may want to precede your apex:outputText with a {!relatedTo.CurrencyIsoCode} merge field.

Wednesday, January 20, 2010

Google Web Toolkit UiBinder Tutorial

Jeff Douglas

I’ve been working on a new project the past couple of weeks that (fortunately) requires Google Web Toolkit (GWT) and I wanted to use the new UiBinder that was released with GWT 2.0 in early December for a number of reasons (clean separation of UI and code, easier collaboration with designers, easier testing, etc ). However, I was having a hard time getting my head wrapped around it given that the GWT site has very little documentation and only a few examples. I’ve combed through the message boards, the docs and the sample Mail application that comes with the SDK and after finally groking the new functionality, I put together a little Hello World app, the kind that would have helped me out originally.

So I’m making some assumptions that you already have the GWT SDK and Eclipse Plugin installed and are familiar with both of them. If you are not, take a look at the GWT site for more info.

To get started, create a new Web Application Project called “HelloUiBinder” in the package of your choice but do not check “Use Google App Engine”.



Now create a new UiBinder template and owner class (File -> New -> UiBinder). Choose the client package for the project and then name it MyBinderWidget. Leave all of the other defaults. When you click Finish the plugin will create a new UiBinder template and owner class.



Open the MyBinderWidget.ui.xml template and add the following code. With GWT you can define your styles either in your template where you need them or externally. I’ve added a small style inline that adds some pizzaz to the label. Notice the field name myPanelContent in the template. You can programmatically read and write to this field from the template’s owner class. So when the owner class runs, it construct a new VerticalPanel, does something with it (probably add some type of content) and then fill this field with it.

Attributes for the elements (the text attribute in the Label element for example) correspond to a setter method for the widget. Unfortunately there is no code completion to get a list of these attributes in Eclipse when you hit the space bar so you either have to know the setters or refer to the JavaDocs each time. A painful process.





.bolder { font-weight:bold; }








For the owner class, MyBinderWidget.java, add the following code. In this class, a field with the same name, myPanelContent, is marked with the @UiField annotation. When uiBinder.createAndBindUi(this) is run, the content is created for the VerticalPanel and the template field is filled with the new instance.


package com.jeffdouglas.client;

import com.google.gwt.core.client.GWT;
import com.google.gwt.uibinder.client.UiBinder;
import com.google.gwt.uibinder.client.UiField;
import com.google.gwt.user.client.ui.Composite;
import com.google.gwt.user.client.ui.HTML;
import com.google.gwt.user.client.ui.VerticalPanel;
import com.google.gwt.user.client.ui.Widget;

public class MyBinderWidget extends Composite {

private static MyBinderWidgetUiBinder uiBinder = GWT
.create(MyBinderWidgetUiBinder.class);

interface MyBinderWidgetUiBinder extends UiBinder { }

@UiField VerticalPanel myPanelContent;

public MyBinderWidget() {
initWidget(uiBinder.createAndBindUi(this));

HTML html1 = new HTML();
html1.setHTML("Click me!");
myPanelContent.add(html1);
HTML html2 = new HTML();
html2.setHTML("This is my sample content!");
myPanelContent.add(html2);

}

}


Now change the entry point class to look like the following.


package com.jeffdouglas.client;

import com.google.gwt.core.client.EntryPoint;
import com.google.gwt.user.client.ui.RootPanel;

public class HelloUiBinder implements EntryPoint {

public void onModuleLoad() {
MyBinderWidget w = new MyBinderWidget();
RootPanel.get().add(w);
}
}


Now open HelloUiBinder.html and remove all of the HTML content between the </noscript> and </body> and save it. Once you run the application, copy the development URL and run paste it into your favorite supported browser, you should see the following.



Now suppose you wanted to nest a widget inside your MyBinderWidget that did something when a button was clicked. We’ll create a small series of checkboxes that allows the user to select their favorite colors and display them when the button is clicked. Create a new UiBinder called FavoriteColorWidget in the client package. Add the following code to the FavoriteColorWidget.ui.xml template.





Choose your favorite color(s):
Red
White
Blue
Submit




Now add the click handler in the FavoriteColorWidget.java owner class.


package com.jeffdouglas.client;

import java.util.ArrayList;
import com.google.gwt.core.client.GWT;
import com.google.gwt.event.dom.client.ClickEvent;
import com.google.gwt.event.dom.client.ClickHandler;
import com.google.gwt.uibinder.client.UiBinder;
import com.google.gwt.uibinder.client.UiField;
import com.google.gwt.user.client.Window;
import com.google.gwt.user.client.ui.Button;
import com.google.gwt.user.client.ui.Label;
import com.google.gwt.user.client.ui.CheckBox;
import com.google.gwt.user.client.ui.Composite;
import com.google.gwt.user.client.ui.Widget;

public class FavoriteColorWidget extends Composite {

private static FavoriteColorWidgetUiBinder uiBinder = GWT
.create(FavoriteColorWidgetUiBinder.class);

interface FavoriteColorWidgetUiBinder extends
UiBinder {
}

@UiField Label greeting;
@UiField CheckBox red;
@UiField CheckBox white;
@UiField CheckBox blue;
@UiField Button button;

public FavoriteColorWidget() {
initWidget(uiBinder.createAndBindUi(this));

// add a greeting
greeting.setText("Hello Jeff!!");

final ArrayList checkboxes = new ArrayList();
checkboxes.add(red);
checkboxes.add(white);
checkboxes.add(blue);

// add a button handler to show the color when clicked
button.addClickHandler(new ClickHandler() {
public void onClick(ClickEvent event) {
String t = "";
for(CheckBox box : checkboxes) {
// if the box was checked
if (box.getValue()) {
t += box.getFormValue() + ", ";
}
}
Window.alert("Your favorite color/colors are: "+ t);
}
});

}

}


The last thing we’ll need to do is add our new widget to the MyBinderWidget template. Open MyBinderWidget.ui.xml and add the custom namespace reference and the FavoriteColorWidget.


;


.bolder { font-weight:bold; }









Now when you run the application it should look like the following.

Tuesday, January 12, 2010

Upcoming Salesforce.com Spring ‘10 Features

Jeff Douglas

For those that don’t have time to weed through all 171 pages of the Spring ‘10 Release Notes, I’ve pulled out a few of my favorites for your viewing pleasure. I didn’t hit all of the items in the release notes so make sure you pull up the the PDF and check out the goodies in detail.

Entitlement Management – Service Cloud
By far the most detailed new feature at 45 pages! You can now set up entitlement management so that your support reps can verify if your customers are eligible for support, create and maintain service contracts, specify service levels on a per customer basis and enforcement of service levels with time-dependent, automated processes. There is a TON of stuff around Entitlement Management including service contracts and milestones so definitely take a closer look!

Quotes – Sales Cloud
A quote is a record showing proposed prices for products and services created from an opportunity and its products. You can create a set of quotes to show different combinations of products, discounts, and quantities so customers can compare prices. Each opportunity can have multiple associated quotes, and any one of them can be synced with the opportunity. You can also create and email PDF quotes.

Answers – Service Cloud
Answers is the newest feature of the Community application that lets community members ask questions, post replies, and vote whether they like or dislike a reply. It looks somewhat similar to Ideas in such that you assign categories to communities and then enable Answers for the customer and partner portals.

Knowledge Enhancements – Service Cloud
A ton on Knowledge enhancements including new API objects for articles, metadata components for data category groups, describe calls for data categories, SOQL and SOSL filter, field level security for articles, custom report types and access to articles in the partner portal.

SOQL Enhancements

New GROUP BY Clause
SELECT LeadSource, COUNT(Name) FROM Lead GROUP BY LeadSource

New HAVING Clause
SELECT Name, Count(Id) FROM Account GROUP BY Name HAVING Count(Id) > 1

Aggregate Functions (Finally!!!!)
SOQL now supports AVG(), COUNT(), COUNT_DISTINCT(), MIN(), MAX(), and SUM()

Date Functions
A slew of new date functions for SOQL to make your life easier.

Semi-Join and Anti-Join Support For Reference Fields
I ran into the issue all the time when writing queries. Now your subqueries can filter by ID (primary key) or reference (foreign key).

Force.com Apex Code Enhancements
  • The Apex scheduler is now generally available.
  • Callouts Enhancements including increased size limit, output messages with multiple elements (hurrah!!), new baked-in DOM class for parsing and generating XML content and enhanced two-way SSL authentication
  • Limits on the number of items a collection can hold have been removed. However, there’s still a general limit on heap size. What!!??
  • sObjects are now created as Objects and you can create generic sObject collections: Set foo = new Set();
  • I’m don’t this this is what Taggert was talking about during his DF09 presentation, but there are a lot of new logging features to check out on line 121. Check it out! Should make life easier until what Taggert revealed in on the roadmap for debugging is GA.
Visualforce Enhancements
  • enhancedList component now uses Ext 3.0
  • Salesforce.com Stylesheets No Longer Called for PDFs
Force.com Web Services API Enhancements
  • New SOSL Clause for Filtering By Data Categories
  • New Support for API Version in Outbound Messages
  • Numerous new Objects to support Entitlement Management, Answers and Quotes
  • There are also some new and changed API calls
  • Support for Sites objects (Site and SiteHistory)
Force.com IDE
The Force.com IDE Spring ‘10 release is scheduled for February 15th. Not sure of the new features but from what I have heard in the past you should (safe harbor) be able to copy from from SOQL search results.

Complex Data Modeling with Multilevel Master-Detail Relationships
You can now include multiple levels in master-detail relationships. For example, in addition to defining a two-object master-detail relationship, such as Account—Expense Report, you can extend the relationship to subdetail records, such as Account—Expense Report—Expense Line Item. You can then perform operations across the master—detail—subdetail relationship. There are some caveats so check out the details.

Sandbox to Production—Change Sets Beta
Use change sets to move configuration changes via a web interface. New enhancements include clone an existing change set, cross-version upload, delete change sets, and a six month expiration of change sets.

New UI Theme
The new UI theme that was unveiled at DF09 is an org-wide update. The new UI supports IE 7 & 8, Firefox 3.0.x and Safari 3.2.x. Sorry Chrome!

Dashboard Enhancements
  • With custom dashboard tables you can create tables with up to four columns, with column totals for number and currency summary fields.
  • With dashboard finder, you can quickly find a dashboard by typing its name in the search filter. All accessible dashboards matching that text are dynamically displayed in the drop-down list.
  • New visibility option for report and dashboard folders to deny access to a folder for portal users (“This folder is accessible by all users, except for portal users”).
Secure Communication with External Websites
Encrypt sensitive date with certificates and key pairs whenever Salesforce.com communicates with a supported external website. Salesforce.com offers either CA-signed or self-signed certificates.

CTI 2.0 Toolkit
A new toolkit for developers and partners to build more robust CTI adapters for call center users.

Globalization Enhancements
  • Translation Workbench Enhancements
  • Translatable Visualforce Email Templates
  • Increased Maximum Limit for Rules
  • Lookup Filter Beta Enhancements
  • Number of Remote Access Application Authorizations Increased
  • Enable Sidebar Search Auto-Complete Setting
  • Enable Single-Search-Result Shortcut Setting
  • A number of Auto-Complete Enhancements
  • Standard Action Overrides are Packageable – Standard action overrides on buttons and links are now packageable for custom objects.
  • Personalized Email Alerts – You can now set the From Email Address in email alerts to the address of the default workflow user.
Limited Release, Beta and Developer Previews

Adobe Flash Builder for Force.com – Developer Preview
Adobe® Flash® BuilderTM for Force.com is an Eclipse-based IDE for developing Force.com Stratus apps (Adobe Air apps that leverage Force.com logic and data). Stratus applications run seamlessly online or offline while taking full advantage of the security, scalability, and reliability of Force.com.

New Report Builder – Developer Preview
A new visual report builder for tabular reports (with limited functionality).

New Opportunity Page – Pilot
An improved opportunity detail page with highlight panels, recommendations, and drag and drop side tabs. Available after Feb. 5, 2010.

My Domain – Limited Release
You can brand your login and navigation URLs for Salesforce to create something like: https://appirio.my.salesforce.com. It also is more secure as it uses HTTPS.

Patch Updates for Packages – Limited Release
Automatically upgrade your managed packages with patches. A push upgrade is a method of automatically upgrading your customers to a newer version of your package. A package subscriber doesn’t need to do anything to receive the push upgrade. If you are developing managed packages this is a must read.

Rich Text Support – Beta
A new rich tech field type that is production quality but with some known limitation. The max size is 32,000 characters including HTML tags and only .gif, .jpeg and .png images are supported.

Wednesday, January 6, 2010

Calling a REST Web Service (JSON) with Apex

Jeff Douglas

Using JSON RESTful Web Services with Salesforce.com opens up your org to a number third-party integration opportunities (Google, Yahoo!, Flickr, bespoke, etc.). JSON support isn't baked into the Force.com platform but Ron Hess at Salesforce.com has created a JSON parser which will do the heavy lifting for you.

Last month I wrote a blog post and example of how to call a REST Web Service with Apex that returns and consumes XML. It was my intention to do the same demo using JSON, however, I ran into a small sang. I couldn’t get the Apex JSONObject parser to work. I tried on and off for a couple of days but couldn't beat it into submission. I checked around the twitter-verse and no one reported much success using the JSON parser with complex objects. I finally cried "uncle" and called Ron and asked for help. Ron was extremely responsive and over the course of a couple of days we worked worked through some of the parsing issues and finally updated the Google project with the changes.



I put together a small demo where you enter your address and the Apex code fetches the address and coordinates from the Google Maps . The service returns the data as a JSON object. You can run this example on my Developer Site.

To get started, you'll need to download the JSONObject class and install it into a Developer org or Sandbox. Unfortunately there is no documentation for the parser so you have to extrapolate from the json.org website.

You'll also need to sign up for a Google Maps API key in order to use their geocoding service. I would also recommend that you take a look at the docs for Google Maps geocoding service.

Here is the Controller for the demo. The interesting stuff is in the getAddress() and toGeoResult() methods. In getAddress(), the user-entered address is used to construct the URL for the GET call to the geocoding service. Make sure you properly encode the address or you may receive undesirable results returned from Google. One thing to point out is line #58. Google is returning a line feed in their JSON response which causes the JSON parser to choke. I simply replace all like feeds with spaces and that did the trick. Ron was going to look into making this change to the JSONObject class in the near future.

I was also having some problems with the geocoding service so I hard-coded the returned JSON object for testing. I checked around and it seems to be a common problem that the Google Maps API randomly returns 620 errors when overloaded. You might want to take a look at the JSON response returned for the hard-coded address. I will give you a little insight for the parsing process.

The toGeoResult() method parses the returned JSON response and populates the GeoResult object with the appropriate data. I chose this Google Maps example because it shows how to parse simple values, nested JSON objects and arrays. The coordinates for the address can either be returned as integers or doubles so I have to check each one.


public class RestDemoJsonController {

public String geoAddress {get;set;}
public String address {get;set;}
public String city {get;set;}
public String state {get;set;}
public Boolean useGoogle {get;set;}

// google api key
private String apiKey {get;set { apiKey = 'ABQIAAAAlI0DHB0p0WGX35GrKEAzQhTwZth5GdZI-P7ekoe_gyhfzl1yZhRAYdM-hb7aEWu30fGchcvGuwuUqg'; } }

// method called by the Visualforce page's submit button
public PageReference submit() {

if (address.length() == 0) {
ApexPages.addMessage(new ApexPages.message(ApexPages.severity.ERROR,'Address cannot be blank'));
}
if (city.length() == 0) {
ApexPages.addMessage(new ApexPages.message(ApexPages.severity.ERROR,'City cannot be blank'));
}
if (state.length() == 0) {
ApexPages.addMessage(new ApexPages.message(ApexPages.severity.ERROR,'State cannot be blank'));
}

if (!ApexPages.hasMessages())
geoAddress = getAddress(address,city,state);

return null;
}

// call the geocoding service
private String getAddress(String street, String city, String state) {

String json;

// hard-coded returned JSON response from Google
if (useGoogle) {
json = '{ "name": "1600 Amphitheatre Parkway, Mountain View, CA", "Status": { "code": 200, "request": "geocode" }, "Placemark": [ { "id": "p1", "address": "1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA", "AddressDetails": { "Accuracy" : 8, "Country" : { "AdministrativeArea" : { "AdministrativeAreaName" : "CA", "SubAdministrativeArea" : { "Locality" : { "LocalityName" : "Mountain View", "PostalCode" : { "PostalCodeNumber" : "94043" }, "Thoroughfare" : { "ThoroughfareName" : "1600 Amphitheatre Pkwy" } }, '+
' "SubAdministrativeAreaName" : "Santa Clara" } }, "CountryName" : "USA", "CountryNameCode" : "US" }}, "ExtendedData": { "LatLonBox": { "north": 37.4251466, "south": 37.4188514, "east": -122.0811574, "west": -122.0874526 } }, "Point": { "coordinates": [ -122.0843700, 37.4217590, 0 ] } } ]} ';

// call the geocoding service live
} else {

HttpRequest req = new HttpRequest();
Http http = new Http();
// set the method
req.setMethod('GET');
// generate the url for the request
String url = 'http://maps.google.com/maps/geo?q='+ EncodingUtil.urlEncode(street,'UTF-8')+',+'
+ EncodingUtil.urlEncode(city,'UTF-8')+',+'
+ EncodingUtil.urlEncode(state,'UTF-8')
+'&output=json&sensor=false&key='+apiKey;
// add the endpoint to the request
req.setEndpoint(url);
// create the response object
HTTPResponse resp = http.send(req);
// the geocoding service is returning a line feed so parse it out
json = resp.getBody().replace('\n', '');

}

try {
JSONObject j = new JSONObject( json );
return toGeoResult(j).toDisplayString();
} catch (JSONObject.JSONException e) {
return 'Error parsing JSON response from Google: '+e;
}

}

// utility method to convert the JSON object to the inner class
private GeoResult toGeoResult(JSONObject resp) {

GeoResult geo = new GeoResult();

try {

geo.address = resp.getValue('Placemark').values[0].obj.getValue('address').str;
geo.keys = resp.keys();
geo.name = resp.getString('name');
geo.statusCode = resp.getValue('Status').obj.getValue('code').num;

// set the coordinates - they may either be integers or doubles
geo.coordinate1 = resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[0].num != NULL ? resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[0].num.format() : resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[0].dnum.format();
geo.coordinate2 = resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[1].num != NULL ? resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[1].num.format() : resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[1].dnum.format();
geo.coordinate3 = resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[2].num != NULL ? resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[2].num.format() : resp.getValue('Placemark').values[0].obj.getValue('Point').obj.getValue('coordinates').values[2].dnum.format();

} catch (Exception e) {
// #fail
}

return geo;
}

// inner class
private class GeoResult {

public Set keys;
public Integer statusCode;
public String name;
public String coordinate1;
public String coordinate2;
public String coordinate3;
public String address;
public String toDisplayString() {
return address + ' ['
+ coordinate1 + ', '
+ coordinate2 + ', '
+ coordinate3 + '] - Status: '
+ statusCode;
}

}

}


The Visualforce page is fairly simple and presents the user with a form to enter their address. If the geocoding services is experiencing issues, the user can check "Use hard-coded Google JSON response?" and the Controller with use the hard-coded JSON response instead of making the GET call to the geocoding service. Once submitted, the address is processed and the outputPanel is rerendered with the resulting address and coordinates.













This example calls Google Map's geocoding REST service (JSON) with the address
you provide below.



Sometimes the geocoding services stops responding due to service availability. If you are receiving errors
with the returned JSON object, you can check the "Use hard-coded JSON response" to use a returned JSON
response hard-coded into the controller from Google's address.





Address






City






State







Use hard-coded Google JSON response?
















Unit Testing

Writing unit tests for callouts can present a challenge. Scott Hemmeter has a really good article entitled Testing HTTP Callouts which should provide you with some useful techniques. You should also check out An Introduction to Apex Code Test Methods on the developerforce wiki.

Tuesday, December 22, 2009

Performing Lookups and Transformations with Talend

Ward Loving

In this article, I’d like to explore the workhorse of the Talend Open Studio – the tMap component. As you’d expect, this component allows you to visually map fields from an input source to fields on an output stream. It also gives you the ability to do lookups to other input files or databases while performing transformations on your data with basic java expressions. We’re going to manipulate some text files to give you a feel for this component’s capabilities, but before we dive in let’s discuss a scenario or two where you might need to use the tMap component:

Use Case A: You’re on a space walk doing some routine maintenance and you look up and see that your toolbox is slowly drifting out of reach into geosynchronous orbit.

Use Case B: You have some data that you need to move from one system to another. Three individual address fields in the source data need to be consolidated into a single text area field in the destination system. You also have a requirement to ensure that only valid email addresses are to be included the output files, but you don’t want to lose a whole record if an email address is invalid.

Sadly, data migration skills are probably not going to help you much with Use Case A. You can skip this article. But, with Use Case B, you’re in business. We’ll start where we left off in the TEST TALEND project that I created in the previous article in June. We won’t be reusing the code from the previous example but you might want to walk through the steps of setting up the repository and creating a new project if you haven’t done that before. You can find and download the latest Talend Open Studio by going to http://www.talend.com.

Open up the Talend Open Studio and select a project. Right-click on the Job Designs folder in the Repository tab and select Create Job. Go ahead and name the job – I called mine Second Job.

Fig 1 – Create Job



I’m am working quite a bit with data migrations to Salesforce.com these days and although you can pull data directly from Salesforce using Talend, I often find myself using Talend to work with files rather than using it to pull data out of Salesforce.com directly. Talend doesn’t allow us (yet), to adjust the number records exported in each batch so it is much quicker to use the Salesforce.com Data Loader to pull records out of Salesforce.com. The difference in performance between the two tools is not a big deal for a few thousand rows – but when you’re talking hundreds of thousands or millions of rows of data, every optimization is helpful. For this example let’s use Talend’s tFileInputDelimited Component which can be used to process the ‘lingua franca’ of Saleforce.com Data Migrations – the csv file. You can find this under File->Input in the components list or you just can just type tFileInput into the Palette Search field.

Fig 2 – Search Palette



Drag the tFileInputComponent onto the workspace.

Fig 3 - Workspace



We’re going to point this tFileInputComponent to a file. You can simply cut and paste the following data into a file called “sample_input_data.csv”:

FirstName, LastName, Email, Address1, Address2, Address3, GenderCode
John, Smith, jsmith@yahoo.com, 666 5th Ave, 19th Floor, Suite 15, M
Jane, Doe, jdoe@gmail.com, 444 8th Ave, Snyder Plaza, Building 7, F
Ralph, Emerson, invalid@email!com, 108 Street,, Apt 3B, M
Edgar, Poe, epoe@scarystory.com, 1000 Boston Street,,, M


For this example, I’ve put the file in my C:\data directory. Now if you double-click on the tFileInputComponent in the workspace it will bring up the Basic Settings of the Component tab at the bottom of the workspace.

Fig 4 – Component Tab – Basic Settings



I’ve highlighted the following settings in Fig 4 above which need to be updated:
  • Select the File Name for the sample_input_data file.
  • Select a comma, rather than a semi-colon as the field separator
  • Select the csv option checkbox
  • Put “1” in the Header field to designate that your input file has column headers.

We’ll also need to input the schema for our csv file, so click Edit Schema and enter the column headers for your file as shown in the figure below. Clicking the green plus sign adds a column entry and clicking the red cross removes one. There are also some handy controls here to import and export your schema entries. This can save you a lot of time when you are working with the same schema across multiple files. Click OK when you’ve completed the schema entries.

Fig 5 – Schema Dialog



Now switch to the Advanced Settings panel and click the “Trim All Columns” checkbox. As far as I can tell, there is almost no reason why you wouldn’t want to trim the fields of your csv files. Making it a habit to check this will save you a lot of headaches performing matching against your lookup files.

Fig 6 – Component Tab – Advanced Settings



You can also change the name of your component in the workspace on the View panel. I’ve changed the label for my tFileInputComponent to “Sample Input File” by updating the Label Format field.

Fig 7 – Component Tab - View



Next we’ll create a simple lookup file to process our Gender Codes. Cut and paste the following data into another file called “gender_lookup.csv”:

Code,Value
M,Male
F,Female


We’ll need to repeat the steps shown in Figures 3 – 7 for the second tFileInputComponent which is pointing to the gender lookup file:
  • Drag the tFileInputComponent onto the workspace
  • Update the Basic Settings – File Name, Separator, CSV Option, and Header
  • Create the schema for the Lookup file. It will have two columns: Code and Value.
  • Check the “Trim all columns” checkbox on the Advanced Tab
  • Update the component to a more user-friendly name (Optional). I’ve named mine “Gender Lookup”

Now in addition to input, our migration process is also going to have output. Let’s stick with the delimited file format for now and create another csv file for the output document using the tFileOutputDelimited component. It can be found under File->Output section of the Component Palette. Go ahead and drag the tFileOutputDelimited component out into the workspace.

Fig 8 – Component Tab – Basic Settings – tFileOutput Component



The steps here to update a tFileOutputComponent aren’t that different from the tFileInputComponent:
  • Select the File Name for the sample_output_data file.
  • Select a comma, rather than a semi-colon as the field separator
  • Select the Include Header checkbox to designate that you’d like your output file to have column headers.

We’ll also need to update the schema for the output component. Remember we have a requirement to consolidate the address fields from three separate fields down to one, so our schema is going to change a little from the input file version.

Fig 9 – tFileOutput Schema Dialog



Click ok when you’ve finished entering your schema. Now we’ll go to the advanced settings panel and click the “CSV Options” checkbox to ensure that any wandering commas are properly escaped.

Fig 10 - Component Tab - Advanced Settings – tFileOutputComponent



Switch to the View panel to update the name of the tFileOutputComponent. I’ve updated mine to “Sample Output File”. Your workspace should look something like this.

Fig 11 - Workspace with Input and Output Components



All our efforts have been in preparation for this moment -- so don’t blow this big guy. If you can take the pressure, and I know that you can, select the tMap component from the component palette under the Processing header and drag it onto the workspace.

Fig 12 – tMap



The crowd roars. That wasn’t so bad now, was it?

The next step is to create the flow for our job. Right click the Sample Input File component and select Row->Main as shown below.

Fig 13 - Create Flow



Connect the little plug to your tMap component. You also need to do the same thing with your Gender Lookup tFileInputComponent. When you right click and select Row->Main from the tMap to the Sample Data Output component the following dialog will pop up. I’ve named the flow “toOutputFile” as shown below. It is possible to create multiple outputs for each job but we just need one for this example.

Fig 14 – Output Name Dialog



A second dialog will come up asking if you want to get the schema of the target component. You do. When you’ve done all this your job should look something like this:

Fig 15 - Workspace with Connected Components



Double-click on the tMap component to bring up the Map Editor. Your input fields are on the left side of the screen and your output fields are on the right. To map one field to another simply drag the field from the left to the right side of the screen and drop it into the Expression column.

Fig 16 – Mapping Fields



After matching row1.FirstName to toOutputFile.FirstName you’ll see a yellow arrow marking the association. Match up the LastName field as well.

You can concatenate all three address fields together by simply dragging each of them to the single address field on the output side of the map editor.

Fig 17 - Address Mapping



Unfortunately, this turns out to be a little crude. If one of the address fields is blank, then you end up with unsightly gaps in your address. In Salesforce.com most street address fields are text areas, so it’s nice to preserve the address lines in the text area which are captured in the source data. Clicking the button labeled ‘…’ next to the toOutputFile Address field in the tMap Editor to brings up the Expression Editor. The button is marked with a red arrow in Figure 17.

Fig 18 - Expression Editor



Cut and paste the following code into the expression editor for more adult address processing:

(row1.Address1 != null && row1.Address1.length() > 0 ? row1.Address1 : "") +
(row1.Address2 != null && row1.Address2.length() > 0 ? "\n" + row1.Address2 : "" ) + (row1.Address3 != null && row1.Address3.length() > 0 ? "\n" + row1.Address3 : "")


This code simply checks to see if a value is present before adding a linefeed to the address line entry. Click Ok to update the expression for your Address field and return to the Map Editor.

To filter out invalid email address we can add a regular expression to the Expression Editor. Click the ‘…’ button next to the Email field and enter this expression:

(row1.Email != null &&
row1.Email.matches("^[\\w\\.-]+@([\\w\\-]+\\.)+[A-Za-z]{2,4}$") ? row1.Email.toLowerCase() : "")


This regular expression isn’t a perfect email validator, but it keeps out most of the riff-raff. If an email doesn’t match the regular expression, it probably isn’t a valid address and we’re just replacing the invalid text with an empty string. An Email type field in Salesforce.com will actually bounce records which have an invalid email format when you load them to the system, which may not be the desired behavior. We’ve only scratched the surface of the capabilities of the TOS Expression Editor. It has all kinds of string, numerical, and date processing functions available to you. The most frequent idiom that I use is a simple java ternary operator. In fact, we could use this idiom to process our gender codes if we wanted to:

(row1.GenderCode != null && row1.GenderCode.equals("M") ? "Male" :
     (row1.GenderCode != null && row1.GenderCode.equals("F") : ""))


This code simply resolves the GenderCode to “Male” or “Female” if the code equals “M” or “F” and returns an empty string if it doesn’t match either one. But let’s resolve our gender codes in more stylish fashion using a lookup. The simplicity of this example belies the power of this capability. It is possible to resolve foreign key relationships in this way between files with hundreds of thousands or even millions of records. Talend’s lookups are more powerful than VLookups in Excel as a way to create foreign key relationships for data migrations and are much more repeatable. There is no limit to the number of lookups you can include. My personal record is eleven in a single job. Although Talend has the capability of writing temp files to disk for large lookups (that is the button that looks like a disk drive on the row2 header in the Map Editor), I have found that you can run into memory issues if your data input has several million rows and your lookup file has several million rows as well. There’s only so much heap to go around. The way I’ve worked around this limitation is to divide the input file into parts using a separate Talend job, and then iterate through the parts using a tFileList component. You can check the Talend component documentation to get more information about tFileList configuration.

To create a lookup relationship between the Gender Code and the Gender Lookup file, drag the
row1. Gender Code field down to the Expr. Key column of the row2.Code field. You should see a purple arrow marking the association, or is it mauve? You know, these are things that keep me up at night.

Fig 19 – Lookup Mapping



Next, drag the row2.Value field to the toOutputFile.Gender field on the right side of the editor window. That’s all there is to it. Your complete mapping should look like the following.

Fig 20 – Completed Mapping



If you click the Inner Join checkbox on the header of the row2 editor, shown in Fig 19 above, only row1 records that have matching gender code will be written to your output file. Click Ok to close the Map Editor and let’s run this bad boy. Select the Run tab at the bottom of the screen and press the Run button.

Fig 21 – Run the Second Job



Check your output file to see the results. It should look something like this:

FirstName,LastName,Email,Address,Gender
"John","Smith","jsmith@yahoo.com","666 5th Ave
19th Floor
Suite 15","Male"
"Jane","Doe","jdoe@gmail.com","444 8th Ave
Snyder Plaza
Building 7","Female"
"Ralph","Emerson","","108 Street
Apt 3B","Male"
"Edgar","Poe","epoe@scarystory.com","1000 Boston Street","Male"


We’ll you haven’t retrieved a $100K grease gun from the abyss, but you have added a couple of tricks to you data migration toolkit. I encourage you to download the Open Studio and try it out.

Wednesday, December 9, 2009

Experimenting with Windows Azure

I've been experimenting with Windows Azure the past few days. It's great to see C# again :) and Visual Studio is such a great IDE. I went through the tutorials and built the Guestbook application from the documentation. There are two basic roles in the Azure Cloud Service. First, there's a web role project that is basically an ASP.NET (C#) application defined to run on the Azure Cloud. Secondly, there's a worker role project in the solution that handles the backend processing for the application. In this case, the worker role is resizing images to thumbnails. The web role and the worker role pass tasks between each other using a queue (Microsoft.WindowsAzure.CloudQueue.Client). This was an easy way to separate the front end code from the back end code. The web role can continue to handle user requests without waiting for the image to be processed.

Web Role: ASP.NET application accessible via HTTP / HTTPS endpoint and is commonly the front-end for the application.
Worker Roles: Background processes for the application.

You can have one or multiple web and worker roles within the same Azure service.

Basically, the example works like this...

  • User posts message and path to image
  • Web role grabs image and stores it in blob storage
  • Web role puts job on queue to resize image to thumbnail (image is rendered in the form in normal size)
  • Worker role checks queue for new messages until it finds the instruction to resize the thumbnail
  • Worker role resizes image
  • Web role refreshes page (image is resized via partial refresh)

This was a great starting point for learning Azure. It made use of the blob storage, table storage, and queues. I'm going to update the example and mirror the Guestbook post to insert data into Salesforce.com. I'll post the source code and the URLs when that's complete.


Google Groups added to Google Apps

Google Apps Premier and Education users will notice the addition of Google Groups to their available services this morning. Groups is hugely popular among communities, user groups, and even closed internal discussion forums but remained outside of the Google Apps portfolio. To overcome this, companies like Appirio, that use Google Groups for internal discussions have their users create identical Google accounts using their Google Apps username. We can now roll Groups right into our Google Apps domain. The following Google Groups features are now included in Google Apps Premier and Education Editions:

  • Fast set-up. Employees and students can now create collaborative groups instantly without burdening IT, and manage the group settings to fit their needs.
  • Searchable archives. Group discussions are archived by default, allowing users to easily search and view past and present discussions via the web.
  • Sharing with a group. Once a group is set up, employees and students can easily share a document, spreadsheet, presentation, shared folder, site, calendar, or video with that group. No need to type in individual email address manually, or remember who joined or left the group. Plus, the shared items will only be accessible by the appropriate people, even as people join and leave the group.
  • Reply on behalf of a group. In addition to communicating via email or the web interface, the new functionality lets group managers send a message on behalf of a group.
  • IT capabilities. IT administrators still manage if and how users can create groups from the administrative control panel.

Here's some quick instructions on how to extend your domain for Groups. Note, this will only work for Premier and Education users. First, login to your Apps dashboard. http://www.google.com/a/cpanel/domainname

Next to Service Setting, click the Add more services... link.

That's it! From your dashboard you can drill back into the console for the Groups service and set your preferences from there. You have the option to make your Groups public or internal, grant access to those who can create Groups, and restrict emails from outside your domain. All the same options you have with Groups today. Have fun... while you're adding services check out Google Apps Short Links(labs) and Google Moderator (labs, but a great example of a Google App Engine application)... if you don't have a Premier of Education version, but want to see more detail on the new offering, check out the short video below.


 
2006-2010 Appirio Inc. All rights reserved.
Appirio.com | Support | Resource Center | Contact | Careers | Privacy Policy