신경망 공부하면서 궁금한 것들 셀프 질답 2

1. 손실 함수(Loss Function)를 코드로 구현하면?

(1) Softmax

def softmax(X):
    exps = np.exp(X)
    return exps / np.sum(exps)

(2) Stable Softmax

def stable_softmax(X):
    exps = np.exp(X - np.max(X))
    return exps / np.sum(exps)

(3) Cross-Entropy

def cross_entropy(X,y):
    """
    X is the output from fully connected layer (num_examples x num_classes)
    y is labels (num_examples x 1)
    	Note that y is not one-hot encoded vector. 
    	It can be computed as y.argmax(axis=1) from one-hot encoded vectors of labels if required.
    """
    m = y.shape[0]
    p = softmax(X)
    # We use multidimensional array indexing to extract 
    # softmax probability of the correct label for each sample.
    # Refer to https://docs.scipy.org/doc/numpy/user/basics.indexing.html
    #indexing-multi-dimensional-arrays for understanding multidimensional array indexing.
    log_likelihood = -np.log(p[range(m),y])
    loss = np.sum(log_likelihood) / m
    return loss

(4) Delta-Cross Entropy

def delta_cross_entropy(X,y):
    """
    X is the output from fully connected layer (num_examples x num_classes)
    y is labels (num_examples x 1)
    	Note that y is not one-hot encoded vector. 
    	It can be computed as y.argmax(axis=1) from one-hot encoded vectors of labels 
        if required.
    """
    m = y.shape[0]
    grad = softmax(X)
    grad[range(m),y] -= 1
    grad = grad/m
    return grad

 

 

2. 활성화 함수를 코드로 구현하면?

(1) Step Function

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
""" 일반적인 step_function """
def step_function(x) : 
    if x > 0 :
        return 1
    else : 
        return 0

""" numpy의 트릭을 사용한 step function 구현 """
def step_function_np(x) : 
    y = x > 0
    # boolean을 int로 변환하면 True = 1, False = 0이 된다.
    # numpy 배열의 type을 변경할 때 .astype을 쓴다.
    return y.astype(np.int)

(2) 시그모이드 함수

def sigmoid(x) :
    return 1 / (1+np.exp(-x))

(3) ReLu 함수

def relu(x) : 
    return np.maximum(0, x)

 

 

3. Gradient란 무엇인가?

쉽게 말하면 미분값이다. 기울기. '구배'라고 한다면 공간에 대한 기울기다. 벡터 gradient는 함수값 y가 커지는 방향을 가리키며, gradient의 반대 방향은 최소값을 향해 간다.

 

 

4. 행렬 미분?

https://datascienceschool.net/view-notebook/8595892721714eb68be24727b5323778/

Data Science School

 

 

5. 소프트맥스 + 크로스엔트로피 손실함수 = Softmax with Loss 계층

https://ratsgo.github.io/deep%20learning/2017/10/02/softmax/

Softmax-with-Loss 계층 · ratsgo's blog

 

 

참고 자료

Classification and Loss Evaluation - Softmax and Cross Entropy Loss

Deep Learning - Activation Function 구현

구배(gradient)란 무엇인가?