Transcript Document 7109630

ACM Multimedia 2004
12th Annual Conference, October 10 -16, 2004
New York City, Columbia University
A Novel Log-based Relevance Feedback Technique
in Content-based Image Retrieval
Steven Chu-Hong Hoi & Michael R. Lyu
Department of CSE
The Chinese University of Hong Kong
Shatin, Hong Kong SAR
{chhoi, lyu}@cse.cuhk.edu.hk
1
Outline
Introduction & Motivation
Log-based Relevance Feedback
Soft Label Support Vector Machine
Experimental Results
Conclusions and Future Work
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
2
Introduction
Content-based Image Retrieval (CBIR)
– Attract much interest, studied for many years
– An important component in multimedia retrieval
– Query based on low-level visual content: color, texture,
shape, etc.
QBE
Challenge: the semantic gap between low-level features and high-level concepts
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
3
Introduction
Relevance Feedback (RF) in CBIR
– A powerful technique, attack the semantic gap problem
– Using interactive mechanisms, soliciting users’ interactions,
learning users’ high-level concepts
– Boosting retrieval performance effectively
– Many popular techniques: MARS, QEX, MindReader,
Optimizing learning, SVM (active), Boosting, etc.
Problems
– Regular relevance feedback techniques: a lot of times of
feedback which will cost much time and make users boring
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
4
Motivation
Relevance Feedback
?
Users’ Feedback Logs
Problem
Can users’ feedback logs information be used to
improve the regular relevance feedback?
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
5
LRF: Log-based Relevance Feedback
Problem Formulation
– Construct a Relevance Matrix: RM
– Each log session: (N =
+
)
N images are marked:
relevant &
irrelevant instances
– Values: relevant (+1), irrelevant (-1), unknown (0)
Image samples in the image database
1
-1
1
-1
-1
0
1
-1
-1
1
1
-1
1
-1
-1
-1
0
-1
-1
1
-1
-1
Log Sessions
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
6
Log-based Relevance Feedback (cont’d)
Relationship Measurement
– For each given session k , if the image i is marked as
‘relevant’ (positive) and the image ‘j’ is marked as
‘irrelevant’ (negative), then the elements are represented as
RM (k, i) = 1 and RM (k, j) = -1
– For every two images: i and j, their relationship
can be
measured by a modified correlation function:
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
7
LRF Algorithm
Collection of training Samples
– Regular relevance feedback
Learn only with a limited number of training samples
Cannot achieve good performance without enough training samples
– Idea: finding more samples based on N initial samples
– For an initial positive sample i, the relevance degrees
between
every image sample j of the database are computed by a soft label
function:
– By ranking the soft label values, we can collect a number of samples
with larger soft label values corresponding to the sample i.
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
8
LRF Algorithm (cont’d)
The learning issue of the algorithm
– Based on the initial marked samples and the log
information, we can collect a large number of positive and
negative training samples associated with soft labels which
represent their confidence degrees.
– Problem: how to develop the algorithm to learn the data
associated with soft labels ?
Proposed Solution: Soft Label Learning
– Soft Label Support Vector Machine (SLSVM)
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
9
SLSVM: Soft Label Support Vector Machine
1
Problem Formulation
1
1
SVM
0.5
1
1
ACM Multimedia 2004
1
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
10
SLSVM (cont’d)
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
11
SLSVM (cont’d)
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
12
Experimental Results
Datasets
–
–
–
–
Images selected from COREL image CDs
20-Category: 2000 image instances
50-Category: 5000 image instances
Each category contains a specific semantic meaning
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
13
Experimental Results (cont’d)
Image Representation
– Color Moment
9-dimension
– Edge Direction Histogram
18-dimension
Canny detector, 18 bins of 20 degrees
– Wavelet-based texture
9-dimension
Daubechies-4 wavelet, 3-level DWT
9 subimages are selected to generate the feature
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
14
Experimental Results (cont’d)
Log Format
– Define a Log Session (LS) as a basic log unit, that
corresponds to a relevance feedback round
– Each log session contains 20 images marked by users
Log Collection
– Collect logs from 10 users
– Non-noisy logs: 100 LS
– Noisy logs:
20-Category: 103 LS, 7.2% noise
50-Category: 138 LS, 8.1% noise
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
15
Experimental Results (cont’d)
Compared Schemes
– EU (Euclidean distance - baseline)
– RF_QEX (QEX: query expansion)
Multiple instance sampling, pick N nearest samples recursively
– RF_SVM
Regular relevance feedback by SVM
– LRF_QEX
Similar to RF_QEX, but we pick the samples weighted by soft
labels in our framework (the larger the label, the smaller the
distance)
– LRF_SLSVM
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
16
Experimental Results (cont’d)
Settings
– Same Kernels: e.g. RBF kernel
– Evaluation metric:
Average Precision = # of relevance / # of returned
– Automatic evaluation:
Taking average precision over 200 query executions
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
17
Experimental Results (cont’d)
Performance Comparison
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
18
Experimental Results (cont’d)
Performance Comparison
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
19
Experimental Results (cont’d)
Performance Comparison
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
20
Conclusions
In this paper we proposed a new scheme to study
users’ feedback logs for improving the performance
of regular relevance feedback in CBIR.
We introduce the soft label learning concept and
developed a modified SVM technique, i.e. Soft Label
SVM, to construct the algorithm for log-based
relevance feedback.
We evaluate our proposed method compared with
traditional techniques and demonstrate promising
results.
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
21
Limitations & Future Work
The proposed LRF with SLSVM algorithm still suffers
performance drop when many noisy logs are appeared.
Much noise may be involved when the scale of the image
database is increased.
When the number of log sessions is large, the dimension of the
relevance matrix may be a problem.
Training time of SLSVM need be considered for large scale
datasets.
Open questions:
Can we work out more effective Soft Label Learning techniques
in the future?
Can we include some noise filtering techniques into our
framework?
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
22
Thank You!
Q&A
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
23
References (part)
[He & King 2003] X. He, O. King, W.-Y. Ma, M. Li, and H. J. Zhang. Learning a semantic space
from user’s relevance feedback for image retrieval. IEEE Transactions on Circuits and Systems for
Video Technology, 13(1):39–48, Jan. 2003.
[Huang & Zhou 2001] T. S. Huang and X. S. Zhou. Image retrieval by relevance feedback: from
heuristic weight adjustment to optimal learning methods. In Proceedings of IEEE International
Conference on Image Processing (ICIP’01), Thessaloniki, Greece, Oct. 2001.
[Hong & Huang 2000] P. Hong, Q. Tian, and T. Huang. Incorporate support vector machines to
content-based image retrieval with relevant feedback. In Proc. IEEE International Conference on
Image Processing (ICIP’00), Vancouver, BC, Canada, 2000.
[Rui & Huang 1999] Y. Rui and T. S. Huang. A novel relevance feedback technique in image
retrieval. In Proc. ACM Multimedia (MM’99), pages 67–70, Orlando, Florida, USA, 1999.on
Image Processing (ICIP’00), Vancouver, BC, Canada,
[Tong & Change 2001] S. Tong and E. Chang. Support vector machine active learning for image
retrieval. In Proceedings of the ninth ACM international conference on Multimedia, pages 107–118.
ACM Press, 2001.
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
24
Appendix
Kernel Comparison
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
25
CBIR
ACM Multimedia 2004
Hoi & Lyu: A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval
26