Tuesday, April 22, 2008
I'll start from the bottom line: If you create your Data Warehouse and you follow the DW rules, you're life will be easy (assuming you know the semantics and the way to build a good and correct DW).
In our case, if you have a dimension and you want to make a Parent-Child hierarchy, your life will be easy if you built the dimension's table in the right way.

For example, let's look at a simple Time dimension:

Time_KeyDayMonthQuarterYearLevel_Num
01012008
01
01
1
2008
1
30122008
30
12
4
2008
1
01/2008

01
1
2008
2
12/2008

12
4
2008
2
Q1/2008


1
2008
3
Q4/2008


4
2008
3
2008



2008
4

As you can see, this time dimension's table contains the days 01/01/2008 and 30/12/2008 and their parents in the levels: month, quarter and year.
Now, let's say that I need to take this dimension and make it a Parent-Child table. This is very simple. Just create a view with one new column which will be the parent column. This is the new column's code (in Pseudo-SQL):
if (level_num = 1) then Month + "/" + Year
else if (level_num = 2) then "Q" + Quarter + "/" + Year
else if (level_num = 3) then Year
else null

The result:

Time_KeyDayMonthQuarterYearLevel_Num
Parent
01012008
01
01
1
2008
1
01
30122008
30
12
4
2008
1
30
01/2008

01
1
2008
2
1
12/2008

12
4
2008
2
4
Q1/2008


1
2008
3
2008
Q4/2008


4
2008
3
2008
2008



2008
4
null

This is it. Now, in your OLAP DB, just configure the new column as the parent and you have a Parent-Child hierarchy. In Analysis Services you even don't have to create a view. In the Data Source View you can add a named calculation and put your code there.

See? When you create your DW according to its rules, the life is easier. In this example, if you created rows for every level in the hierarchy and created a descriptive key - everything is great.
This can help you in many scenarios. For example, when you find that your dimension is not balanced then you might want to make it a parent-child, so you won't have many pseudo-levels when the only relevant member is the leaf. Otherwise, this can be very annoying to your user. In SSAS, make sure you don't use it too much, because it is bad for performance.
Wednesday, April 23, 2008 4:06:32 AM (Jerusalem Daylight Time, UTC+03:00)

One more thing about getting a file from the web/SharePoint and using it as a source in SSIS: If you need to authenticate just change the xml.open command to:

xml.open "GET", URL, false, "user", "password"

where user and password are the user & password that has permissions to the desired file. Note that it is VERY recommended to have an application user, so the password won't be changed in the future. If you don't have such user and you must change your password in the future, do not forget to change it in the script. My tip: add a reminder in your calendar to change the password in the script.

In this point I don't know if you can authenticate using SSL or stronger protocols using VB script.

Tuesday, April 22, 2008 10:20:05 PM (Jerusalem Daylight Time, UTC+03:00)
 Monday, March 31, 2008
We got many client requests for the ability to show in their web sites the "last updated" date of the data.
It doesn't matter how you show the data of the SSAS - the customers will always want to know for which date the data is true.
My solution includes a ASP.NET 2.0 web site that uses the AMO class libary. It takes the date from the server and shows it to the user.

What you need to do is:
1. Open a new ASP.NET web site using Visual Studio 2005/8.
2. Add the AMO dll (Microsoft.Analysis Services). You'll find it in the SSAS server.
3. In the already-made default.aspx page, just add one Label.
4. Add a configuration file which will hold the name of the SSAS server. That way, when you install the site from the development environment to the production environment, you'll only have to change this file. Call this file config.xml and write in it the following:
<?xml version="1.0" encoding="utf-8" ?>
<ServerName>YourServerFullNameHere</ServerName>

5. In the code-behind file (default.aspx.cs) write the following code instead of what you already have there:

using System;
using System.Data;
using System.Configuration;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;
using AMO = Microsoft.AnalysisServices;
using System.Xml;

public partial class _Default : System.Web.UI.Page
{
  protected void Load_Page(Object sender, EventArgs e)
  {
    Label1.Text = GetCubeUpdateDate(Request.QueryString["DBName"],Request.QueryString["CubeId"]);
  }

  private string GetCubeUpdateDate (string dbName, string cubeId)
  {
    using (AMO.Server asServer = new AMO.Server())
    {
      asServer.Connect("Data Source=" + GetAnalysisServerName());
      AMO.Database db = asServer.DataBases.FindByName(dbName);
      if (db == null)
      {
        return "DB Name not found";
      }

      AMO.Cube cube = GetCubeById(cubeId, db);
      if (cube == null)
      {
        return "Cube Name not found";
      }

      DateTime lastProcessed = cube.LastProcessed;
      return lastProcessed.Day.ToString() + "/" + lastProcessed.Month.ToString() + "/" + lastProcessed.Year.ToString();
    }
  }

  private string GetAnalysisServerName ()
  {
    XmlDocument xmlDoc = new XmlDocument();
    xmlDoc.Load(Request.PhysicalApplicationPath + "config.xml");
    return xmlDoc.GetElementsByTagName("ServerName").Item(0).InnetText;
  }

  private AMO.Cube GetCubeById (string cubeId, AMO.Database db)
  {
    foreach (AMO.Cube cube in db.Cubes)
    {
      if (cube.ID.Equals(cubeId))
      {
        return cube;
      }
    }
    return null;
  }
}

Eventhough the code is self-explained, here are some points referring it:
  • I chose not to include the server name in the web.config file because I like to seperate application-related configuration and web configuration.
  • If you want you can get the cube name from the user (in the query string) and then the code is even shorter - just get the cube like I got the database.
  • I wanted to show the date in the format DD/MM/YYYY, so that's why I did the long return statement in the GetCubeUpdateDate method. If you want to return the date in the MM/DD/YYYY format you can use the lastProcessed.GetShortDateFormat() method.
  • Note that when you publish the web site you need to create a dedicated virtual folder in the IIS.
  • The user uses this site in the following way: All he need to do is to create a frame with this site's address as its source and add it the DBName & CubeId in the query string. In SharePoint it's even easier - the uses only need to create a page shower web part.
enjoy.

Monday, March 31, 2008 11:05:20 PM (Jerusalem Daylight Time, UTC+03:00)
 Saturday, March 29, 2008
Two weeks ago I showed you the leds map. This time I'll describe how it is done.

The leds map is basically a web page with a lot of java-script and Panorama applets which together bring the user a feeling of Ajax & DHTML based web site.
Look at the picture of the map in the previous post. The leds are simply Panorama applets which show Panorama views. Each view shows only one led. Using the Panorama SDK, I did the following:
  • Take the led's value from the view and show it in the tooltip
  • Take the led's color from the view and let the user filter the map according to the desired color(s)
  • Take the led's view path and after clicking the view, show the related views (the departments' views)
The rest of it is just java-script games and tricks.
The leds map is a beautiful example of what you can do with imagination, thought and good will to give your customer a good working BI tool to work with.

 | 
Sunday, March 30, 2008 5:27:29 AM (Jerusalem Daylight Time, UTC+03:00)
 Wednesday, March 19, 2008
Yesterday, my friend Ilya asked me how to perform average for dates. I explained him that actually, the dates in SQL Server are represented as numbers, where zero is 01/01/1900. All you need to do is to cast the dates to numbers, make average on them and getting the result back as date. Assuming that the date column called MyDate, Here is the code:

Cast(Avg(cast(MyDate as float)) as datetime)

Thursday, March 20, 2008 12:04:23 AM (Jerusalem Standard Time, UTC+02:00)
 Sunday, March 16, 2008
A month ago I posted about the necessity of sharing ideas in the BI world. I really think that if we all share our smart ideas then we'll be better in our work.

I want to show you a work I finished few weeks ago. I'm very proud of this work as it will be in the desktop of our CEO and I got many compliments for it.
Note that what you see in the picture is not the real screenshot of the work (It's much more beautiful in the reality...). There's a problem getting out screenshots out of my company, so I did a sketch in Power Point.

This is the functionality of the leds map (my design, if you have any comments):
  • The leds map is simply a web site, meaning zero-footprint in the client's computer. Some computers in my company has java compatibility problems, so I added a parameter you can send with the site's URL which changes the applet's java version (see more in the next post, which will be more technical).
  • The leds map has to be small, about a quarter of the screen. That's because it's intended to be a part of the CEO's desktop.
  • When the map loads, a picture with a turning-around The Thinker statue is shown with a "Loading" message below (our CEO loves that statue...).
  • After the map has been loaded, the user sees two axis with the leds in them. The two axis can represent any Meta-Measures you'd like: Short-Term Profit Vs. Long-Term Profit, Client's Satisfaction Vs. Company's Profit, etc. This is a point that many people have difficulty to understand, so I'll give an example: The yellow led is in the top-right corner, so that says that the underlying measure is very important in both the meta-measures. Going on with the example, that says that this measure is very important for theClient's Satisfaction and for the Company's Profit. Note that the leds never move. Only their color changes.
  • When you move the cursor on a measure in the map, a small tooltip appears next to it. The tooltip shows the measure's name and its value (You can see it in the left-bottom led). Design Change: As my team master recommended, now each led has its measure's name above it. The tooltip shows only the value.
  • When the map loads, only the red leds are shown. In the top-left corner of the screen, there's three radio buttons which filters the shown leds by their colors. In the picture, all the leds are shown because all the radio buttons are enabled.
  • Clicking on a measure on the map drills-down to the different department's leds, as you can see in the left side of the picture.
  • Clicking on a department's led makes the map to vanish and instead of it there's a drill-down of the department, meaning that the measures of its sub-departments are shown instead of the map.
  • After the last drill-down was made, there are two possible actions: Close the new view and return to the map or open the new view in full screen, where you can slice-and-dice and play with the data.
In the next post I'll describe how the leds map was built using the Panorama SDK.

 | 
Sunday, March 16, 2008 7:55:46 AM (Jerusalem Standard Time, UTC+02:00)
 Saturday, March 15, 2008
It figures that using simple excel file as a source in SSIS is not so trivial, especially if your source is in the web or in your SharePoint portal. At first you'll think it's easy - just declare the excel source as a url (the url of the excel file, for example) and it will succeed. The problem is that Microsoft let you think it's working. Click on the excel source and you'll see in its properties that the source path is the local temporary internet files, meaning that the source is a local copy which is not up-to-date, so it's worth nothing.
Here's what I tried to do and the final (and successful solution):

1. Use the File System task. It won't work because you can't declare an URI there.
2. In the MSDN forum (I can't find the link right now) they say to write a script, so I also tried this. Using the Script task, I written a code in VB.NET which using the System.IO libary of the .NET framework and copies the excel file (using its URI) to the desired location in the local computer. Running it, I got an error saying that the script can't use URIs...
After trying this I understood that every code or action running in the SSIS context won't work with URIs. I'm not sure I know why Microsoft developers built it that way (or maybe it's just another bug). Anyway, the next step is the solution.
3. Build an executable file that performs the desired copy task. You can't use regular batch (.bat) file because DOS/CmdExec does not know how to work with URIs. So, there are two ways to perform this:
a. Download this and use it as a copier from the web.
b. Use the following code and save it as a Visual Basic Script file (*.vbs):

'GetRemoteBinaryFile.vbs
TheFile = "myExcelFile.xls"
DestFolder = "C:\SSIS_Sources"
URL = "http://mySite/myFolder/myExcelFile.xls"
Set xml = CreateObject("Microsoft.XMLHTTP")
xml.Open "GET", URL, False
xml.Send
set oStream = CreateObject("Adodb.Stream")
Const adTypeBinary = 1
Const adSaveCreateOverWrite = 2
Const adSaveCreateNotExist = 1
oStream.type = adTypeBinary
oStream.open
oStream.write xml.responseBody
' Overwrite an existing file
oStream.savetofile DestFolder & TheFile, adSaveCreateOverWrite
oStream.close
set oStream = nothing
Set xml = Nothing

After you have your file (vbs or exe) you can use the Execute Process Task in order to make the copy. In the task, declare that you want it to run your exe or vbs.  After that, just use a normal Data Flow Task, where the source excel file is in the local computer (the file that was copied in the previous task) and the destination is your desired DB.

Note that:
1. Before executing you must have the excel file already placed in your local computer, meaning that you must make the first copy before the first time you run the package. This is because SSIS performs integrity check before running the package and it checks that the file exists.
2. Even if the copy process is long (because it's coming from the web), don't worry. SSIS work synchroniously, meaning that the Data Flow task will not start until the Execute Process task which copies the file will end.

Enjoy.

Update: I added a post about authentication.

Sunday, March 16, 2008 6:47:26 AM (Jerusalem Standard Time, UTC+02:00)
 Sunday, March 09, 2008
I can't sleep. Just can't fall to sleep. I can't stop thinking about the terror attack that occurred last Thursday. For those of you who don't know or can't remember - you can look here. I would like to show this story from my personal point of view.

"Merkaz HaRav", or as we all call it - "Merkaz" (Center) is the main Yeshiva (aka Jewish Seminary) in Israel of our way. This terror-act was well planned and well performed. Hitting Merkaz is not just another terror-act. Merkaz is the main Yeshiva of us all, meaning the jewish people who believe in the Rabbi Kuk's way - conjoining the holy and the secular, the holy books and the military, the religious brother with the non-religious brother (we all brothers, first of all) and so on. All that happened in the middle east  for the last years began in this yeshiva: The settlements beyond the green line, knowing that we own this land who was given to our ancestors. The resurrection of the jewish people in our holy land and the eternal search for rightness and peach.
Hitting Merkaz meaning hurting us all, hitting our ideology, hitting everything. We all feel (we all talked about it, of course) that this attacked just happend in our homes. When the heart is hurt you'll feel it in the entire body. I just can't understand how our government can proceed like this.

I intended to paste here some pictures from the scene, but I don't want to shock you (look here if you really want to). The CNN and the international news companies don't show you these pictures and especially the pictures of the joy of the little children in Gaza strip (which I can't find right now). Little kids, full of hate, who are happy in the murder of others.

May g-d help us all.

Sunday, March 09, 2008 7:15:16 AM (Jerusalem Standard Time, UTC+02:00)
 Monday, February 25, 2008
update: Damn, the title of this post just seems bad...

I think that after almost two and a half years of fatherhood (hey, I just invented a word!) being a parent I can write some things about my experiences. But first, a short introduction from the world of Operation Systems:

Thrashing (Wikipedia) is a case where one process in the computer is consuming huge amount of memory for a long time and therefore makes the whole system very slow. There are many approaches of solving this problem (Yeah, you guessed right: Windows doesn't treat this at all). The two important solutions (for this post) are:
  • Kill the memory-consuming process (Used in Linux 4.2)
  • Make this process "the king" of the computer: Let it run until it will end (Used in Linux 4.6)
These approaches can also be used with children. When a children is misbehaves you can either give him a loud "No" (No, I don't mean to kill him. A kid is not a process :-)) or let him be the king - just let him do what he wants. Now comes the important question: Which is the right approach? Well, I think that there's no ideal answer. It's just depends on the situation: Does the children right about what he wants? Maybe I (or his mother) promised him something earlier? What is the educational result of my action? All these things may change the answer.

The most important of all: Do what is right for your child. Not what is comfortable for you in the moment.

Monday, February 25, 2008 11:59:20 PM (Jerusalem Standard Time, UTC+02:00)
With Panorama SDK you can do cool stuff as I will show you in the future. Though, There are important (and undocumented) things you must know before you start. A very common task is to change the shown view views. Note that:

If you load the view using the Parameter "Alias" with the full view path (ends with  ".xml") you won't be able to change the view. later. Worse: The applet will not return an error. It will just won't respond. So, if you want to enable the dynamic change of the view, in the "Alias" parameter only enter the name of the Briefing Book where the desired view is. Next, add another parameter named "FirstView" and there enter the relative path of the view, meaning that you'll have to remove the name of the server and the briefing book's name. Don't forget to replace the back-slashes (\) in double-back-slashed (\\), otherwise... the applet won't respond. Some examples:

use: AttachParameter("Alias", "http://<myServer>//<ThePanoramaDirectoryPath>//<myBriefingBook>//<myDirectory>//myView.xml"); to show a view with no option to change it later (not recommended). Note that here you don't need to use back-slash because this is just a regual URL.

use:
AttachParameter("Alias","myBriefingBook");
AttachParameter("FirstView","\\<myDirectory>\\myView.xml");
to show a view with an option to replace it later using the CallShowView function.

I recommend always using the second method. That's because you can't know what will be the next demand of your customer. Remember that this is a very common thing to do in the BI world.

 | 
Monday, February 25, 2008 11:39:22 PM (Jerusalem Standard Time, UTC+02:00)
My team master always says that Oracle was left behind in the BI world because they don't have good visual tools over the OLAP cubes. Microsoft, for example, has good visual tools such as Panorama. Today I've seen that they have beautiful visualations over relational DBs which called ADF Data Visualization Components. The problem is that these tools can't look over OLAP cubes. In this link (look for Oracle OLAP Q&A) you can see what we'll see from Oracle in the future: These visual components will be able to show OLAP data. Maybe that will make Oracle really be able to fight Microsoft in the BI scene.

Monday, February 25, 2008 11:07:02 PM (Jerusalem Standard Time, UTC+02:00)
I had a big test yesterday in the university, so now I can blog again.

the title says it all. Enter here if you're a student. MS offers many free developers tools for students under a project called DreamSpark. Among the tools you can find: Visual Studio 2008, Windows Server 2003, Game Studio 2 (which I eager to learn for a long time, especially because I'm a gamer) and the known Express editions of SQL Server and Visual XXX (you name it).

Enjoy.

Monday, February 25, 2008 10:48:54 PM (Jerusalem Standard Time, UTC+02:00)
 Sunday, February 10, 2008
My boss called me today to ask me if some things can be achieved using our technologies (mainly Analysis Services and Panorama NovaView). These things were simple ideas of how BI can be shown to the end users. As we talked I thought of many great ideas that can be done. As you well know, one of the biggest problems in the IT world is that the user doesn't always know what he wants. Drilling down to the BI world, I can say that the problem is that the user can't dream. He can't know what he can ask for and sometimes - how easy it can be achieved. One of our many tasks as BI consultants/designers/developers is to help them dream. We can show them things that we've done and things that others have done. This is where you, the reader, can help.

Let's share ideas. In your blogs (or as comments in blogs) you can write about beautiful things you did in your organization. This can be idea or real UI that you can show. Don't worry, I'm not only-talking man. In the near future I will show here something very beautiful (and big ROI of course...) that I did. Stay tuned.

Monday, February 11, 2008 4:38:22 AM (Jerusalem Standard Time, UTC+02:00)